Brain surgery on our oven

I normally write about cloud computing and infrastructure, but occasionally I do other things.  Very occasionally.   I do enjoy tinkering with things and fixing things, though.  Actually, more to the point, I enjoy figuring out how to fix things more than actually fixing them.  (This means that anything that breaks a second time typically gets neglected).

Anyway, our oven developed an “interesting” (read “very annoying”) problem.  All of the buttons would work except the temperature up button.  When you press “temperature up”, the oven thinks you want to do a timed cook.  Guess which one of these functions I need more?

So, for a few weeks, everything in our house has been baked at 350 degrees (the default temperature) or below.  We couldn’t raise the temperature.  Needless to say, this caused some strife in the Dotson household, so I did a little research and ordered a new control panel.  That will arrive in a few days.  In the meantime, however…

I pried the glued-on membrane control panel off the front.  (I’ve already had to work on a different part of this oven, so I knew that there were no high voltages anywhere near this part.  So everyone please Don’t Panic that the oven is still powered.  In fact, that was really helpful for diagnosing the issue.)  After getting the stupid adhesive to let go, the membrane panel came up and looked like this:

 

Now, the way these things work is that there’s a “matrix” of connections.  Imagine a numeric keypad where the buttons are in neat rows and columns, with wires for each row and each column.  When you hit the 1 key, the first row and first column are connected.  When you hit the 6 key, the second row and sixth column are connected.  I stole this image from Google images to illustrate, so if you want a keypad, buy it from this place:

So, getting back to my oven, here’s the interesting part of the keypad (if you’re still reading this, you hopefully won’t snort at my use of the word “interesting”):

I added a blue dot on top of the temperature up button that I’d really like to work, and a red dot on the stupid “Stop Time” button that keeps getting triggered instead.

My oven panel is just like the keypad above, except that the keys are spread all over the place so the rows and columns are hard to see.  You have to follow the traces to see which two connections each button is hooked to.  However, if you stare at it for a few minutes, you can see that there’s a spot where a trace from the bad button “jumps over” the trace from the good button.  I drew a red rectangle around that in the picture above.  That’s gotta be the problem — those must be shorted together!  It’s just my bad luck that if the controller thinks you’ve pushed both the temperature up and the Stop Time button at the same time, that it should go into the Stop Time routine.  Priorities, oven, priorities!

I figured maybe some moisture had gotten in there and shorted them together, so I tried some canned air to dry it out, but it was sealed fairly well so that didn’t help.  (This is where having power to the oven was great, because I could try the buttons to see what was going on.  It was about this time that I figured out that the button I thought was getting pressed incorrectly wasn’t actually the one that was getting pressed, because the good button and bad button in that case didn’t share any traces.  When you think the universe is being inconsistent, check your assumptions!)

Well, if I can’t fix that “jump”, what could I do?  I could certainly do without the “Stop Time” button.  Hmm…  I followed the trace beyond the “bad” button and found that there’s only one other button connected to that “row”, which is the Upper Oven button.  (This is a double oven).  I don’t have to have the Upper Oven button, because if you start doing things it will assume that you want the upper oven anyway.  So, if I can break that trace, then I can use my temperature button at the expense of the Upper Oven and Stop Time buttons.  That sounds like a good deal.

So, while the poor oven was on and completely aware, I took my drill and drilled through that trace at a convenient place before the “jump” (see the red lightning bolt above for where I drilled).  Now, I can use my temperature up button!

The only problem is that if you set a temperature on the upper oven, then push lower oven and set a temperature there, you’ve got no way to get back to controlling the upper oven unless you turn both of them off and start over.  Also, I drilled a little bit too far and made a tiny dimple on the front panel, but hey, I’ll replace it in a week or so anyway.  The panel’s glue still held when I pressed it back on, so the only evidence of this brain surgery is the little dimple below the timer set button:

Some people on the Internet have wired up microswitches instead of a membrane keypad on appliances.  I salute them, but I’m lazy and that was more work than I felt like doing.  However, when I get the new controller board in, I might experiment with using a wire and some conductive glue to route over the bad “jump”.

StorEdge 2015

I thought I knew quite a bit about enterprise storage.

I studied computer science in school; we learned a lot about L1 and L2 cache, cache hit/miss ratios, and the amazing differences in speed between L1, L2, DRAM, and disk.  And years later, we saw the revolutions in storage area networks, virtualized storage, and flash.  Only a few years ago, I ran fairly large data centers with over a petabyte in enterprise storage units using SAN Volume Controllers.

But you know how it is in the computer industry; you get distracted for just a moment and the whole world changes.  It’s as if you’ve got these kids named Cloud, Network, Security, and Storage, and you turn your head for a minute to yell at the other three and when you look back, Storage has paint all over the wall.  That’s how I felt listening to Dr. Axel Koester and Clod Barrera at the Storage Systems Technical Kickoff at IBM Edge 2015, except that Storage has painted a really nice picture in this case.

It seems as though SSDs just got here, but Axel and Clod told me that they’re now obsolete.  Note that this doesn’t mean that flash technology is obsolete – but packaging up the flash chips in a little box that looks like a hard drive and having the computer pretend that there’s still a spinning metal platter inside there is not the way to go.  With SSDs, you only get 50 gazillion times more speed than hard drives, and when using flash directly, you get 200 gazillion.  I may be slightly off on the numbers.

To me, the lines between cache, ram, and disk are all blurring.  We still have volatile memory (the stuff you wish you’d saved when the power goes out) and non-volatile (the stuff you wish you’d backed up), but beyond those distinctions it’s getting hard to tell.  (My security personality feels compelled to point out that even “volatile” memory might last longer than you think with the power off — google “cold boot attacks” some time).

Some systems have four levels of cache, two different types of RAM (fast and slow), and now varying speeds of “disk” (which may be flash or spinning platters).  And beyond that, on the network you have fast object storage and slow object storage (“nearline”), and really really slow storage which may take minutes to retrieve a piece of data but is extremely cheap.  There are charts, papers, and algorithms to help you pick the right storage for your particular workload.

On the slightly different topic of data density, this slide made me giggle.

Apparently, while you can read extremely narrow tracks of data, the write heads plough a huge, wide swath through the magnetic media — like me trying to use the riding mower where I should be using a weed trimmer.  One solution researchers have worked on is to write one layer, then the next mostly overlapping that (so that only a small part remains readable), then the next mostly overlapping that, and so on like roofing shingles.  The next time I look up, Storage will be off to college.

Hybrid Cloud Security at IBM Edge 2015

I am an IT infrastructure architect, so in that role I tend to think about accomplishing business goals in terms of components such as storage, servers, and network. I’m also a cloud architect, so I tend to think about accomplishing business goals using a combination of different cloud services. And, I’m a security architect, so I tend to think about designs that keep the bad guys out of the systems, whether they are private systems, or public clouds, or both – hybrid cloud is the new normal. I don’t think I’m unusual in this respect; we all use many different tools to accomplish our goals, and we all need to think about security in our solutions.

For many organizations, on-premises infrastructure certainly isn’t going away any time soon, if ever. There is plenty of existing infrastructure that has many years of useful life left. There are some infrastructure requirements that aren’t met by “off the shelf” public cloud offerings. There are occasionally security requirements that are not met in a shared environment, no matter how strong the virtual “walls” are. And, there’s always the pesky speed of light to consider; for example, if you have an on-premises workload, you will often have performance problems if the storage is 20 milliseconds away in an off-premises cloud environment.

The IBM Edge conference is about infrastructure innovation, and that means not only talking about the components, but also how to integrate your internal infrastructure with cloud offerings to solve the biggest problems quickly and securely. It’s amazing how fast this industry moves; technologies such as Virtual Storage Center now make it possible to automate tasks such as “right-tiering” that previously took hundreds of hours of manual, error-prone work. Storage virtualization allows you to swap out storage units without the servers and applications even realizing that you’re upgrading. Servers and networks just keep getting faster. And all of the infrastructure components keep accumulating new features that make management easier, or reduce the amount of time tasks take, or make the system more secure.

Encryption is a feature of particular interest to my security “personality”. Encryption technologies are being built into many different layers of solutions, and as time goes on, I expect to see many different layers of the solution bring their own encryption. Encryption is not simply a check box; each layer that encrypts can guard against different attacks. Disk encryption, such as offered with the DS8870 line, can act as another control against loss due to improper data sanitization, improper data disposal techniques, or theft. Encryption on the wire can protect against eavesdroppers, who can easily use either software or hardware based methods to watch the data going over the network. Encryption at the operating system level can guard against offline attacks against the disks. Encryption at the middleware or application layers can guard against an attacker elsewhere on the system being able to access data. These techniques are important in an on-premises data center, and are even more important in hybrid cloud environments!

In addition to encryption, there are many other controls to make hybrid cloud systems more secure. I’m presenting at IBM Edge on Hybrid Cloud Security – I hope you’ll come see me!

Episode 10: A New Home

I’ve been blogging off and on for several years at IBM’s ThoughtsonCloud blog.  ThoughtsonCloud recently changed formats, and my older posts are no longer visible on the main site.  Hey, that’s the Internet for you — if it’s more than a few minutes old, it’s not worth looking at any more.

In case you aren’t living on Internet time right now, and you’d like to view any of my old posts, they’re still available at http://archive.thoughtsoncloud.com/author/cdotson/ .  If you’re so inclined, you can also view my tweets at https://twitter.com/crdotson , and any comments I make on Linkedin at https://www.linkedin.com/pub/chris-dotson/0/b55/6a9/ .