Building Retro Dither: Dithering Algorithms, Run-Length Encoding, Resource Forks and More!

Today I launched a new novelty Mac app, Retro Dither. Retro Dither gives any photo a cool retro look using just black and white pixels. You may want this for artistic effect, or you may want to export your photo to MacPaint for display on a retro Mac. Retro Dither launched on the Mac App Store today.

Note that we’re talking about just black and white, not grayscale (no grays here!). Amazingly, dithering algorithms make pure black and white images look like grayscale. You’ll get an old-school newspaper look, or a look like your photo just came off of a Mac from the 1980s. And this works with any photo in any bitmap format (JPEG, PNG, HEIF, etc.). Retro Dither can then export your dithered image to MacPaint format for display on black and white Macs going all the way back to the 1986 Mac Plus.

This is the story of how I developed Retro Dither…

I was working on my next programming book, which will be an intermediate Python projects book, when I came across an article about Atkinson Dithering on Hacker News by John Earnest. Dithering algorithms can be used for approximating the look of an image with less colors. Atkinson Dithering is one that is particularly well suited for approximating an image using just black and white. I was in the middle of developing a project for the book related to computational art and I was thinking I would need another art project to fill out the chapter. And Atkinson Dithering really caught my eye. First, because it looks cool. Second, because I was a Mac user growing up and I remember all the software and games that actually used it.

But there was a problem. After reading John’s article, I found the algorithm too simple to sit alongside the other projects I was developing for the book. I can explain the gist of it to you in a few sentences. You take each pixel in an image one at a time from the top left of the image to the bottom right. You check if it’s closer to black or white. Whichever it’s closer to, you turn it into (if we stopped there, we’d have a simpler dithering algorithm known as Threshold and it wouldn’t look great). Then you take the difference between black or white and the original pixel and call that the “error.” You spread that error into some of the pixels to the right and below the original pixel. Then you continue to the next pixel. John explains it in more detail in his article.

Then I got an idea. Bill Atkinson, the creator of Atkinson Dithering, was also the creator of MacPaint, a program I loved growing up. What if we not only dithered the image, but then we exported it to MacPaint format? That would be really cool and would add a bit more heft to the project. So, I started researching the MacPaint file format and found this article. It’s not a particularly complicated file format, but it at least includes another useful algorithm, Run-Length Encoding, a simple compression scheme where instead of writing out repeated sequences like “AAAAAAAAA” we essentially would write 2 bytes that say “9 As”. Now we have two simple, but interesting algorithms. That’s perfect for a project in the book.

I had to learn a bit more about the MacPaint file format. For example, it encodes pixels at 1 pixel per bit before it does the run-length encoding on the pixel-packed bytes. 0 is for white pixels and 1 is for black pixels. It runs the run-length encoding on a per-scanline basis. Every MacPaint file contains exactly 720 scanlines and 576 pixels per scanline (basically a 576 x 720 resolution). I found my hex editor, Hex Fiend, really helpful when debugging. I would look at my output MacPaint files and compare them to example MacPaint files, including some I still have saved from my childhood. I also found an open source project that converts files the other way (MacPaint -> modern bitmap). I could run my program’s converted MacPaint files back through it and see if the same image came out the other end.

With the help of Pillow for reading the initial bitmap image for conversion, I was able to build the whole thing in a couple hundred lines of Python. Again, this was the perfect size project for the book, so I was feeling really good about it. Then I tried my MacPaint files in actual MacPaint and there was a problem. They wouldn’t open!

Thinking back to my early days in computing, I remembered about type and creator codes. These were metadata that the Classic Mac OS (versions 1 through 9) used to associate a file with the program that should open it instead of using a file extension. The problem was my files had none. I used a Classic Mac program to add them back in and my images successfully displayed in MacPaint! But this wasn’t an ideal workflow. I wanted my users to be able to just double-click their finished files. So, I wanted to add the type and creator codes into the exported MacPaint files from my Python program.

The problem was, the Classic Mac OS stored files in two separate sections, known as data forks and resource forks. It stored the type and creator codes in the resource fork. Almost every other operating system, including modern macOS, Linux, and Windows doesn’t have resource forks. Files are just a single blob. And my research was showing me that there would be no easy way to add a resource fork to my file for transport so that the Classic Mac OS would see it as one file. I needed a work around.

Of course tons of people had the same problem before. Classic Mac OS users had to work with people on other operating systems. So transport file formats were developed that could combine data forks with resource forks together in one file for transport and then “unbundle” them on Classic Mac OS so they could be used. The simplest and most common (other than the later developed .sit, Stuffit archives, basically Zip files in Mac land), was MacBinary.

Turning my MacPaint files into MacBinary files for transport with correct type and creator codes was as simple as adding a 128 byte header to the file in MacBinary’s specified format. This is still not a perfect solution, because the files need to be unstuffed by a program on the Classic Mac OS side that understands MacBinary before the MacPaint files can be double-clicked, but programs that understand MacBinary were common on the Classic Mac OS and include the aforementioned and ubiquitous Stuffit.

I got this all working in a couple nights for the book and then I thought, surely someone else must make a commercial MacPaint converter. The retro Mac community is pretty huge right now. But I couldn’t find any, other than the venerable Graphic Converter, which is a full-featured graphics program that I recommend (its creator, Thorsten Lemke, was actually gracious enough to let me conduct an interview with him for a research project in college back in 2006) but that costs $30.

I decided to make something lighter weight. Thus, Retro Dither was born. I ported my Python code to Swift. I added 4 more dithering algorithms (they’re all similar to implement). I built an AppKit frontend. I tested it. I tested the output files in emulators like SheepShaver for MacPaint 2 on Mac OS 9 and Mini vMac for MacPaint 1.5 on System 6. I did find a couple bugs, like that MacBinary expects each fork to be in exactly 128 byte chunks (I just had to fill extra space with some 0s). Even though later versions of Stuffit will ignore this, other MacBinary unbundlers like the built-in MacBinary command-line app on modern macOS and the original MacBinary program from the the standards authors in the 1980s will not (by the way, the modern macOS’s Archive Utility will unbundle MacBinary files).

I hope you enjoy Retro Dither. And if you want to be made aware of my next Python book when it comes out, which will include the project described here, please join my very low volume mailing list.

Apple Apologetics

The common criticism of the modern news industry is that objective journalism has been replaced by opinion-slanted proselytizing. There’s an interesting slice of the news world where proselytizing was never even at issue. I’m talking about the world of news sites, blogs, magazines, and rumor mills that cover Apple.

I’ve been reading news sites like MacNN (now defunct), MacWorld, and MacRumors daily for more than two decades. I also enjoy more personal “news” and commentary blogs like Daring Fireball and Six Colors. And then there’s the journalists like Mark Gurman and David Pogue that at some point in their careers covered tech in an Apple-centric way for real news organizations like Bloomberg and the New York Times. There’s a cottage industry of journalists and outlets that make their entire living on the happenings of Apple Inc. Surprisingly, many of the specialized sites even managed to scrape by when Apple was merely a ten billion dollar company, and not the two trillion dollar behemoth it is today.

There’s an important assumption about the readers of these outlets. It’s assumed you’re a fan of the company. Why would you read MacWorld or a lot of what David Pogue wrote back in the heyday of his Apple fervor if you didn’t own an Apple product? But being a fan of Apple generally means more than being a user. In the 1990s when, as a kid, I first became a rabid Apple fan, it was known as a cult (in fact one of the news sites is still known as Cult of Mac). Why were you using the non-standard products of this small struggling computer company with less than 5% marketshare? You had to believe in their products’ superiority or the potential of the company’s future despite the clear market signals to the contrary. You had to believe. It was a religion for techies.

That belief in Apple and its products has always been imbued in the outlets that cover the company. The fans demanded it. It was why we kept reading. We wanted our faith to be reinforced despite all of the contrary voices. Reading the news sites was like going to church. It made sense in the context of the ’90s. They were the underdog. It was fun and cool to root for the rebels. And the potential was there. If we could just keep them going with our collective enthusiasm, they could reach their true destiny.

But then they did. They became the largest company in the world. They came to dominate several slices of the tech industry. You’d think that would mean the coverage of them would change. Yet, the cottage industry that covered them never lost its fervor. It’s like the prophecy was fulfilled and now those who believed would go to heaven and be angels, rewarded for their faith.

And angels they are. Towards the company. The new presumption is no longer that some unrealized dream could be achieved when the right products arrive. The new presumption is that the company is a force for good. How did this transformation happen? It used to be about “could they deliver the long promised tech future.” It was about Mac vs Windows. For a while it was about iOS vs Android. Now it’s about social issues, battles with government, and court cases.

I think that we’re all looking for something to believe in. And many of us, including the “journalists” and commentators that cover Apple, are not finding it in the rest of our lives. Yet somehow our belief in this company during its dark times, against all odds, was right. And so we want to continue to believe our faith in it will be right again. Yet, today, we’re being asked to believe in something very different. A different company facing a different set of challenges in completely different spheres. I’m still a big fan, but I no longer take every leap of faith.

Book Review: Sid Meier's Memoir!

I’ve played far too many hours of Civilization throughout my life. If you haven’t, or you’re not an aspiring game designer, then you may find large sections of Sid Meier’s Memoir! hard to appreciate. But I do love Sid Meier’s games and I do dabble in making games as a hobby. So, I really appreciated this memoir that takes you through Meier’s entire career as well as some of his rules of game design that have made him so successful.

This book is not a deep dive. Each chapter, which generally covers a game or two, provides some insight about how a game was developed and why it worked (or did not work) for the player. But the chapters are short, and many of them concentrate on just one aspect of the game or Meier’s philosophy. The two things I found most interesting were the inspiration behind each game, and the gameplay rules that Meier learned over time.

It’s not a tell-all, and it doesn’t cover much about the video game business. You won’t get a play-by-play about the development of Civilization VI. Meier says that he left much of the business side to his partners. Instead, this is a book about the creative process.

Meier touches on enough of his personal life to give you a general outline of his life and some of the real-life hobbies that led to some of his best games. Again, most of the details are left out, but there’s enough there to be inspiring. The thing I found the most surprising in the entire book, is that despite his high position, Meier continued to be an active participant in the programming of his games (or at least the prototypes) late into his career.

In short, Sid Meier’s Memoir! is limited in its scope. It’s mainly about game design and the creative process. But it does that well. You won’t quite feel like you’re in the trenches with the team making Civilization, but you will feel like you have a sense of where their head was at. The lessons about game design and the creative process that Meier provides are drawn from deep experience and most seem self-evidently true. The writing is clear and the insights are real. If you loved his games or you want to make games, you should read this book.

Book Review: No Filter

Instagram is one of the most important cultural phenomenons of the 2010s. No Filter by Sarah Frier covers Instagram’s rise from a two-person startup to the second largest social network in the world. Along the way, Frier explores Instagram influencers, the network’s effect on teens, and its sometimes rocky existance within the confines of its parent company, Facebook. No Filter is competently written and it seems Frier had significant access to the important players. Yet, the narrative lacks a compelling spark, and despite the access, fails to reveal much insight beyond what is already well known about the company.

No Filter is written in the “anonymous amalgamation of interviews” style of other recent hit business books like Hatching Twitter and Console Wars. This means that the interview source of each bit of original reporting in the book is not directly revealed in a footnote or endnote. This is not only problematic for the book as a historical document, it also means the reader is not aware of who’s bias is tilting a particular anecdote. This style can be partially justified if it leads to a more compelling narrative. However, while I found Hatching Twitter and Console Wars to be page turners, I didn’t feel the same way about No Filter. It may just be that Instagram’s corporate history is not particularly exciting compared to Twitter or Sega, but I also think it had to do with the way the book is structured.

The narrative is somewhat abruptly interrupted halfway through to spend a couple chapters on Instagram’s cultural effects, and the people who make their living through the app. Although I am not a regular user of Instagram, I was already aware of basically all of the material in these chapters just from loosely following the news and the cultural zeitgeist surrounding the company. I didn’t feel these chapters added enough value to be worth the interjection during what should have been the heart of the book.

Like many business books written by journalists about software companies, No Filter’s coverage of the technical aspects of Instagram is inadequate. Instagram co-founder/CTO Mike Krieger’s technical leadership is praised, but the reader does not get a sense of what kinds of technical decisions he really made that were successful. What technical hurdles did Instagram face as it scaled? How did they solve them? This is ultimately the story of a piece of software and part of its success is due to the engineering of that software. That angle deserved better coverage.

Speaking of Mike Krieger, we don’t get to know him very well in No Filter. In fact, the only people we get a strong sense of, are Instagram co-founder/CEO Kevin Systrom and Facebook CEO Mark Zuckerberg. It’s like a book about the early days of Apple that talks about Steve Jobs, but only scratches the surface with regards to Steve Wozniak. Okay, Mike Kriger might not quite be a Steve Wozniak, but he probably deserved a few more pages.

Frier does a very good job with Systrom, though. The reader gets a strong sense of his personality. Frier plays up the battles and contrasts between Systrom and Zuckerberg very well. We come to understand that some of the decisions made at Facebook surrounding Instagram were personal, and others were based on the very different characters running the two social networks under the same roof.

When you add up the “influencer” chapters and the lack of character development for many of the players, No Filter starts to feel like a relatively short book. Yes, Instagram is not a very old social network, but given the author’s access, there was surely more to be revealed. I wanted more business, technical, and character narrative and less filler.

Ultimately, No Filter is not a bad book. If you want a cohesive 300 pages about Instagram, you will get it. It’s just a bit of a bland book. It may be that Instagram’s corporate history is not that exciting. Or, it may be that the author missed some opportunities. Either way, I commend Frier for taking on Facebook, a powerful entity in our society that deserves further examination. And her portrayal of the relationship between Kevin Systrom and Mark Zuckerberg was very well done.

Book Review: Ghost In the Wires

Surprisingly, for a memoir about a guy who spends most of his time sitting in front of a computer, Ghost in the Wires: My Adventures as the World’s Most Wanted Hacker is a gripping thrill ride. And it’s a testament to Kevin Mitnick and his co-author, William Simon, that it works. In the wrong hands, the same material could easily be boring or overly technical. They found the right balance of action, technical detail, and non-hacking content to keep the story engaging. The book gives you a real sense of the hacking underworld, as well as how surprisingly easy it is to social engineer individuals, companies, and government agencies.

The most surprising part of the story is how little of Mitnick’s exploits were due to what we traditionally think of as computer hacking, and how much was due to social engineering. Mitnick learned to be a master manipulator, and in this book he explains to you how he convinced cops to turn over records, trusted employees to send files to people they just met, and just about anyone to do anything over the phone. If you want to better protect yourself from social engineering, this book is a great primer. It really makes you think about how quick we are to trust someone with a bit of confidence when they know a couple details we assume they would only know if the confidence were warranted.

Mitnick manages to make himself relatable. By including personal details, descriptions of family life, and imagery of his surroundings, he comes across as a regular guy. He compares his hacking activities to an addiction. I can almost buy that. He was getting notoriety, solving interesting puzzles, and probably feeling the same kind of rush that cleptomaniacs feel. It sounds like it could easily become a compulsion if you’re good at it and don’t have a certain moral wavelength turned on.

And I say that last sentence carefully. Because Mitnick (as far as I know and he claims in the book) never did anything especially damaging compared to some of the other well known hackers. He says he wasn’t in it for money or to do harm, but instead to satiate his curiosity. A significant portion of the book concentrates on this fact, and how the media played up his story to make him sound a lot more evil than he deserved. And that apparently affected his prosecution by the government. John Markoff, a well known New York Times reporter at the time, is singled out for particularly incredulous stories.

Yet, my biggest criticism with the book, would be Mitnick’s lack of sympathy for his victims. He spends a lot of time emphasizing how little damage he did, and almost no time apologizing for the damage that he did do. Sure, he may not have sold the source code he stole for a profit. And sure, the people he tricked mostly just had their time wasted. He didn’t actively try to ruin anybody’s life. Yet, mitigating his “work” inevitably cost companies and individual a lot of time. Employee time is money. It probably cost taxpayers millions of dollars investigating, trying him, and catching him. His exploits made people feel unsafe and caused them emotional distress. And he doesn’t seem very sorry. For that reason, I found him especially difficult to root for during the early chapters. Even his “pranks” as a teenager sometimes seemed mean spirited if I were to be on the receiving end of the frustration they caused.

In the end, though, Mitnick won me over. I found his relationship with his mother and grandmother endearing. I think the way he turned his life around after getting out of prison the last time is remarkable. It seems he’s done a lot of good the last twenty years. He’s an example of why people deserve a second chance, and his book is an interesting examination of social engineering and the media-legal system complex.

The Web Has No Design Standards

A reader recently complained to me about the hyperlinks on this blog. The reader thought the links were too hard to distinguish from the rest of the text. And the reader’s right. The Swedish Greys desktop theme that I thought looked cool eight years ago, while attractive in an aesthetic sense (to me at least), is not the most usable or accessible. I’ll be looking for another theme.

I was able to style my blog however I wanted to and it looks the same in all browsers. That’s the flexibility of good HTML/CSS standards. Every site can look and behave exactly as the creator envisioned. It’s also why the Web’s a usability nightmare. We have to learn to use every site we visit because every site is designed differently. How come when we talk about Web standards the focus is almost entirely on technical standards? Where is the worry about design standards?

I recently finished reading the classic book The Design of Everyday Things by Don Norman and it talks about standards. Normans says “When all else fails, standardize.” Basically when you have no other way of implementing good design, you turn to standardization so at least every user only needs to learn how to use the similar things (in this case web pages) once. And I think we have no other way, because if we did, we would have figured it out in the past 30 years.

It wasn’t always this way. I remember using the pre-CSS and pre-JavaScript Web as a little kid on Mosaic. You knew there that the blue underlined text was always a hyperlink. And you knew that the back button always took you back a page. And you didn’t have to worry what different actions buttons did, because there was no JavaScript. I’m not saying we should go back there, but in many ways having the constraints made pages easier to use. There was no need to think. Now we have no constraints, but that’s why we need standards.

Every other major consumer computing platform but the Web has design standards. Apple’s platforms are famous for their Human Interface Guidelines. They are an attempt to ensure all apps follow some standard design conventions. Not every app does, but Apple has some ability to enforce them through its app stores, and some users even demand developers follow them. So, they are at least kinda sorta followed by most major apps. If the Web had design standards, maybe users would demand developers follow them too. Google and Microsoft have design suggestions and guidelines for their developers. This is why a good app for each platform feels “at home.”

But we have no design guidelines for the Web that are widely accepted. Sure, people have tried. But the only way we’re going to get something that’s actually followed is if we have a standard. And a standard needs to come from a standards body (Apple, Microsoft, and Google are the standards bodies for their respective platforms). W3C, please put some focus on a design standard. Not everybody will be forced to follow it, but it could do a lot of good in terms of usability.

Classic Computer Science Problems in Java is Published

I am pleased to announce that my fourth book, Classic Computer Science Problems in Java, has been published. It is now available for purchase from the publisher’s website. As Manning’s deal of the day, the book is available today (January 5, 2021) for 50% off.

Classic Computer Science Problems in Java is a continuation of the Classic Computer Science Problems series, with previous incarnations in Swift and Python. They teach problem solving techniques from the realm of computer science in an approachable code-centric tutorial-like fashion. They are relatively light on theory and heavier on analogies, examples, and code. You don’t need a computer science education to pick up the books. In fact they were designed with self-taught programmers in mind. You do need to be at least an intermediate programmer. You can find out more about the contents of the series at classicproblems.com.

The Java book follows the success of Classic Computer Science Problems in Python which has sold more than ten thousand English copies and has been translated into eight other human languages including Portuguese, Simplified Chinese, German, Russian, Polish, Korean, Traditional Chinese, and Japanese. Beyond the three original book programming languages, the source code has been ported by the community into five additional languages including Go, C++, Ruby, PHP, and JavaScript.

I don’t know if the Java book will be as successful as the Python book, but I do know that there will be a lot less readers upset about the inclusion of type annotations. Manning provides a short free sample of the book on their website, so you can check it out before you buy. Let me know if you have any questions on Twitter. I’m @davekopec.