Saturday, January 23, 2021

When black boxes seem like voodoo magic

I've been practicing on Leetcode for some time. As some of you may be aware, it is a platform that helps one solves problems online, comes with an online compiler, built in test cases etc, and validates submissions on the basis of given time limits and memory requirements.

While solving one problems on graphs, I hit a roadblock - the test cases just wouldn't pass - test case 3 failed.

Now this was interesting, so I inputted the test case manually to check

Guess what, it worked. Now this is surprising, how can a test work individually when its otherwise failing? So I thought maybe some conditions in my code were not accounting for all execution states, and made minor alterations (no logical change), and resubmitted the code.

But this time, another test failed. And interestingly enough, this was failure at test case 2 (which had passed earlier), which meant that something even more fundamental broke.

Unable to understand what it means, I tried the input again, manually, and it worked:

But, but, but. How is this possible? I finally realised there must be something here which was common in multiple runs, but different in individual runs. Looking carefully at the second error message, it is clear, that some variable was being shared across the runs of the different test cases, when it clearly shouldn't be as per the platform. 

Because, How can an empty list test case find a node for 2?

It seemed, the culprit was here:

Rather than recreating the instance of Solution class, the platform is simply re-sharing the older instance, which contains results as per the previous run, as I've declared the variable visited as a class variable. So the solution was simple, re-instantiate this variable at every run

And we have success:

Thus, what essentially seemed like a bug or Voodoo magic to the lay user, who doesn't have much idea into how the input is being evaluated internally, is simply a reflection of larger phenomenon. End users, and consumers of any black box - be it a an app, a system, a service or a platform, are always at a disadvantage when facing an issue that is unique because of the manner of operations. 

While in this instance, it might have been a feature (or a bug), it is always better to have well intentioned error messages that help the user understand what s/he is doing wrong.

Thus, while the first error message above was not helpful, it was the 2nd one which allowed for some degree of insight into what went wrong here.

No comments:

Post a Comment