Tag Archives: ComicBase

Things I’m Learning as I Learn to Livestream

I recently decided to start up a ComicBase Livestream, so that I (your friendly neighborhood ComicBase creator) could reach out to the folks directly, and answer your questions live through the magic of internet broadcasting.

In the interest of aiding all who are considering setting up their own broadcast, here’s a few of the bits of technical trivia I’ve learned in the past two weeks as a newbie livestreamer:

  • “AFV” means “Audio Follows Video”. It also means that if you push the little button marked “AFV” on your ATEM Mini Pro switcher, you’ll be activating your video camera’s tinny little built-in microphone any time you switch the shot to the camera. The switcher then adds that audio to the sound from the big condenser microphone you had plugged into the switcher’s separate “Mic” port, giving your a bitchin’, doubled-up boomy sound. If you’re a student of audio engineering, you’ll recognize that this as reminiscent of the trick used to record David Bowie’s “Heroes”… except that it sucks.
  • The ATEM Mini Pro switcher is a fantastic piece of gear–basically giving you a portable TV switching console with the ability to directly livestream right from the switcher. But it lacks a headphone jack, so you’ll never hear the boomy AFV thing until you play back the audio after the livestream.
  • The FocusRite Scarlett 2i2 audio interface I sprung for in order to be able to use a decent condenser microphone for the shoot is also wonderful… but it only has one set of outputs. This means that if you have those going to your speakers so you can hear your computer, you can’t also send the output to your video switcher to take in the sound of the condenser microphone you’re using the Scarlett’s phantom power for. For about a day, I fooled around with various Y-cable scenarios, but eventually gave up and realized I just shoulda bought the more expensive Scarlett 8i6 in the first place. Which I then did. If you need a lightly used 2i2, please reach out: it’s going cheap.
  • If you want the big AT-4033 condenser microphone you’ve suspended on a boom stand to not slowly drop from the top of the frame into the center of your desk while shooting, there’s a trick. It is not, as it turns out, to wrench the tightening knob on your boom as hard as you possibly can (won’t work, strips the mechanism), but instead to suspend a small sandbag counterweight on the far end of the boom. Physics: the ultimate Lifehack!
  • My wonderful (and expensive!) Canon 5d Mark III camera will apparently auto-stop recording at 29 minutes, 59 seconds (a “feature” built into most DSLR cameras designed to avoid being classed as a “video camera” for EU taxation purposes). This is less than great if you suspect you might want to talk for 30 minutes or more.
  • On the other hand, the more-than-a-decade-old Canon HF-100 handycam I scrounged from my back closet works beautifully for shooting the video. But it also has an uncanny ability to auto-focus on the boom microphone I intentionally left in the shot, leading the focus on my face to be a little soft. This is flattering for hiding my increasing number of wrinkles, but is not a standard cinematographic artistic choice.
  • The reason almost every YouTube livestreamer except the truly professional begins their streams by staring blankly at the camera is that they’re watching YouTube’s preview window to see if it’s actually picking up their video feed before they click the “Go Live” button to start broadcasting the stream. After clicking that button, YouTube then apparently randomizes the starting frame by +/- 3 seconds in order to capture that uncomfortable staring look you had going as you waited for the video to appear in preview.
  • Live chat is amazing–but it scrolls by really quickly, and it’s nigh-impossible to catch anything but a tiny amount of the discussion until after the stream is over.
  • Despite all this, Livestreaming is a blast! I’ll be continuing to do it each Wednesday at 4pm Central, no doubt making new and creative mistakes each broadcast–and having a great time answering viewer questions, talking about new and obscure ComicBase features, and what have you. Please check out the channel here, and do the old “Like and Subscribe” thing to help spread the word, and get notified of new broadcasts!

Breaking HTTPPostAsync When Debugging in IIS Express, or “Wasting 5 Hours in Programming’s Version of a Really Crummy Escape Room”

It’s 3 AM, the day after Daylight Savings Time threw everyone’s internal sleep clocks into absolute chaos. (I say “chaos” based on both my own personal feelings, as well as the flood of fire service calls we’ve had today, including an overdose, a suicide attempt, and numerous other ways that our local residents have signaled their general lack of fervor at the idea of getting up tomorrow).

Worse yet, had it not been for the time change, I could have started this blog post with “It’s 2 AM, and the fear is gone” — and my opening would have been much cooler. Now I’m blaming Daylight Savings Time for writer’s block too. Way to go, DST.

But nevertheless, here I am, writing a pretty darned geeky blog with the hopes that some poor schmoe might stumble upon it in a session of mad Googling and save themselves some of the five hours I’ve just blown on one of the more painful programming pitfalls I’ve managed to stumble into in recent memory.

As part of a general modernization of ComicBase’s web APIs, we’re testing out a new set of calls to our servers which locate all the items you’ve sold on Atomic Avenue and let you deduct them from your inventory–as well as (minor spoilers here) finding all the comics you’ve scanned with the app while you’re out in the real world and which you now want to add to your desktop database.

Since it’s incredibly helpful to be able to watch the action on both the client and the server side of things when you’re doing work like this (and since it’s considered presumptuous for the programmer to set breakpoints on the production server which would stop the site cold), I’ve been working with a local copy of the ComicBase.com and AtomicAvenue.com sites, running under a development version of the web server software called “IIS Express” . Things had been going well, and I was watching the program carefully validate the user’s credentials, look up their databases, get the right data and post it back to the user–all the while checking for all the jillions of things that could go wrong in terms of bad passwords, invalid user accounts, lost network connections, and just about any other simulated problem you can imagine–trying to make sure we handled them all as gracefully as possible.

It’d been a long weekend on this project, but as I say down around 10 to finish things up, I was feeling pretty good about my chances to knock off early, grab a beer, and maybe even check out that crazy Polish cyberpunk video game I’d started a while back (Observer). All I really had to do was step through the different cases in the debugger, make sure they were being handled right, then remove the breakpoints and watch the whole thing run at speed to get a sense for how the system would feel in real use.

Everything was going well, but as I started tidying up and removing my breakpoints, breakpoints, suddenly I started getting bad data back from the web requests which were rock solid mere moments earlier.

So I put the breakpoints back and started single-stepping through them, puzzled all the server calls came back exactly as expected–only to give 404 errors moments later when I let them run at speed.

That’s when the night started to blur into one long slog which resembled nothing so much as an escape room whose puzzles had been planned by a madman. I’d check the code, it would behave. I’d set a breakpoint for a couple of lines after the call completed, and it’d work. But if there was ever a case where two web calls in a row fired off, the second one would always fail.

“OK”, I thought… It’s probably some sort of thread issue, which seemed all the more plausible that any call that I waited even a couple of seconds on before proceeding to in the debugger would run normally. Unfortunately, chasing down problems like this–whether they’re thread deadlocks or inadvertent calls to non thread-safe libraries–are a royal pain in the tucchus to track down.

The hours went by as I double-checked that all my async calls were properly awaited, that I hadn’t accidentally blocked them by calling “.result” at the end of any methods, and so on and so on with all manner of obscure programming lore. This was followed by endless googling on StackOverflow to see if anyone else had a similar problem or could suggest answers.

I tried removing the asynchronous calls; I tried marking all the relevant async calls with ConfigureAwait (False) to help them keep their context straight; I even tried rewriting all the HTTPClient calls in the old-style WebClient mode which allowed me to get rid of the mere idea of anything being asynchronous at all. Sure it’d mess up system performance and make the app seem slower to users, but as the clock edged past 2 AM and all the Fiddler packet traces in the world showed nothing useful, I was willing to try darn near anything to make some progress.

But even rewriting the whole set of web calls to be fully synchronous using the ancient WebClient routines was getting me nowhere. They ran great in the debugger, but immediately returned 404 errors when run without breakpoints set. What the living heck was going on?

So then–as much to make my Fiddler traces make more sense if I had to post the whole thing up on StackOverflow in the hopes that someone smarter than me could figure it out, I decided to move the new routines up to our production server and get a trace of them running there.

And they worked.

Perfectly.

With no debug points set.

Over the next several minutes, many curses were muttered as I leaned on the Ctrl-Z (Undo) key and watched the last several hours of my typing undone, block by block, until I was basically back where I was when I sat down to work tonight. The only real difference was that the code I was using to call the routines was pointing to the real server, running the real version of IIS instead of the IIS Express running on my development system.

And the whole darn thing was working right.

Sooo… what did we learn here? Well, there’s apparently a strange glitch in the behavior of the various web pieces of the Microsoft web client framework which keeps repeated calls to those routines from resolving properly when used on a Microsoft Visual Studio 2017 session on IIS Express. Basically, if you’re going to use the local server to debug, something may not resolve quite as fast as it should when it comes to the web calls, and if your calls start stacking up, you might want to try either slowing down your debugging, or moving some of the critical pieces to their final homes and testing there before you give up.

I also learned a lot of ways not to solve this problem, which has its own sort of value to programmers. And I would up learning about 4 entirely different techniques for making web post calls–all of which blew up in exactly the same way when run at speed on the development system. In a way, that’s what made me suspect that the problem may not have been purely code-related at all.

I also learned that I truly detest Daylight Savings Time. And now at 3:55 am, I am absolutely going to bed.

“Hey, I’ve got an Idea…!” The Secret Story of Sidekick

“Damn it! It just can’t read the update file any faster!”

It was late 2012, and I had a little bit of freedom after the launch of ComicBase 16 to try to take on some of the “Big Issues” for ComicBase’s future. High on the list was better mobile support (more on this in a separate article), a possible replacing of the underlying database technology, and possibly even facing down the prospect of completely rewriting in .Net.

“Dotnet” as it’s pronounced, is a Microsoft technology that had clearly been the future of the company’s development path for some time–but which promised to pose a monumental struggle for porting the mammoth code base behind ComicBase. We’d actually done an investigation of what it would take to make the move three different times over the years, but had to turn back each time when it became clear we’d have to essentially rewrite and refactor what had become a very large and complex program. Worse, if we somehow managed to rewrite ComicBase in .Net, our developers would get the benefit of much better build tools (albeit at the price of endless hours of programming and re-testing), but the customers would be unlikely to notice any difference at all.

Actually, that last part isn’t quite true: the progress bars in .Net are decidedly nicer. The rest of the changes would be technical and architectural in nature–which is to say, virtually invisible to the end user, unless we used the rewrite as an excuse to polish up various bits of the program using the newer technology.

But for today, I wasn’t worried about any of those things. I had decided that I wanted to see what could be done to make the weekly updates faster.

Introduced in ComicBase 10, the weekly updates did something previously unimaginable: the ComicBase staff had taken on the job of keeping all our customers updated with all new comic information the same week the comics appeared on the stands. We’d supply all the new data on every comic released each week–along with its artists, writers, storylines, and other special information–and our customers would be able to simply download it and have their database be instantly current, instead of adding all that data in themselves. All the customer had to do then was simply check off which books they had in their own collection, or better yet, use a barcode scanner to “bleep” them in to their own collection.

For understandable reasons, the weekly updates were a huge hit, but it meant that we had to take on the incredible amount of work to acquire several hundred new comics each week, as well as keeping up with the constant pricing changes that were happening in the world of comics. We opened up our own Diamond Comics account, and soon were ordering one copy of virtually every issue sold, which our indexers would scan and index within a day or two of their arrival so they could be part of the Friday update.

Later, we added in a “Submit new or corrected data” feature to ComicBase which allowed several dozen amazing customers to add to the wealth of knowledge we were processing, and the pace of additions to the database doubled, then doubled again. Soon we were processing thousands of new issues and additions each week, and the database grew to encompass virtually every English-language comic that had ever been printed–as well as hundreds of thousands of foreign books.

But now there was a new problem: the sheer size of the database was starting to make the process of downloading and processing the weekly updates an increasingly lengthy process for our customers. What once took them only a few minutes was starting to stretch on for 15 minutes or longer–sometimes much longer if they had a slower machine or were upgrading a very old version. If customers hadn’t been updating regularly, it was not unheard of for an update to contain hundreds of  thousands of  updates to everything from pricing to artist credits. Unfortunately, updating this much information meant that customers were spending too much of their time watching progress bars while they waited for all those changes to be incorporated.

So I had decided to take some time and really pound on the code for the updating process, trying to wring the last bit of performance out of it. Numerous late night hacking sessions ensued, but for every clever programming trick I came up with that saved a few ticks of the clock, the time savings soon vanished as the flood of new comics swelled the database to ever greater scope.

After yet another late night of coding, I was discussing the problem with my wife Carolyn as we walked over to Starbucks on our morning routine. “I think I’m at the limit–no matter how fast computers are going, it just looks like it’s going to take several minutes to even read–let along process–the update file. After all, it’s got something over 10 million distinct pieces of information in each one.”

“Can’t you cut it down?” she asked?

“Not that I can see. There’s no telling how long it’s been since someone updated, and we need to be able to catch them up to date even if they haven’t checked for updates all year long. We could cut down on the amount of data we offer, but a big part of the appeal of the program is that it’s the biggest database of comics in the world.”

“If only there was some way to have the updates happen automatically so that people weren’t standing around waiting for them each week–.”

“Hey, I’ve got an idea…”

(to be continued)

Found Inside the Old Loveseat, 1995-2010

Tear apart an old couch with a reciprocating saw, and discover a treasure trove…