Ben's Bloghttps://benhoff.net/2019-12-11T07:50:00-06:00Tensorflow Serving is Dead2019-12-11T07:50:00-06:002019-12-11T07:50:00-06:00Ben Hofftag:benhoff.net,2019-12-11:/tensorflow-serving-dead.html<p>Tensorflow Serving is Dead, or at least not being maintained</p><p>As part of my OpenCV <span class="caps">CVAT</span> development, one of the things that I want to focus on is making it easier to develop computer vision models within the platform. There’s quite a few startups that are also working in this space, but the idea of crowdsourcing labeling/annotation in a single platform and then using that same platform to integrate and train a machine learning model, is powerful. It’s the vision I was working towards with my last company, and still an interesting one.</p>
<p>As part of that vision, the <span class="caps">CVAT</span> platform needs to integrate in some of the top tier machine learning frameworks. <span class="caps">CVAT</span> has already integrated in OpenVINO (which is an Intel Deep Learning framework. Actually hilariously, OpenVINO is rebranded <a href="https://github.com/opencv/dldt"><span class="caps">DLDT</span></a>). But the next obvious enterprise framework is Tensorflow.</p>
<p>One important thing from a systems perspective is to separate out your resources, so you don’t risk breaking your entire system and you can scale different pieces at different rates. So the model integration really needed to be of the microservices brand. Enter stage left: Tensorflow serving.</p>
<p>Tensorflow serving dockerizes models and allows them to be served via <span class="caps">REST</span> <span class="caps">API</span> or gRPC. So you can have a fat <span class="caps">GPU</span> on a separate server from the <span class="caps">CVAT</span> instance, plugging away, training the best inference possible. This is perfect for <span class="caps">CVAT</span>’s use case.</p>
<p>There’s only one slight hangup. Tensorflow has recently transitioned into Tensorflow 2.0, an <span class="caps">API</span> breaking upgrade. This has (as <span class="caps">API</span> breaks always do), thrown the entire ecosystem into chaos. But it has shown some light on an interesting thing.</p>
<p>Tensorflow Serving is Dead. Or at least, not being actively maintained.</p>
<p>See part of the upgrades to Tensorflow 2.0 was to remove the <code>contrib</code> module, a top level module that used a very popular Tensorflow library known as Slim. This breaks Tensorflow Serving’s examples. See <a href="https://github.com/tensorflow/serving/issues/1475">here</a>. No big deal though, software is flexible and can heal right? Well, only if you keep development resources on the project. Two perspective pull fixes, both <a href="https://github.com/tensorflow/serving/pull/1477">one of mine</a> and <a href="https://github.com/tensorflow/serving/pull/1486">someone else’s</a> have gone unmerged or commented for 30+ days. The last pull request accepted was September 11th, 2019.</p>
<p>So Tensorflow Serving is officially dead. Or at least unmaintained at this point.</p>Encouraging Open Source Contributions2019-12-10T21:29:00-06:002019-12-10T21:29:00-06:00Ben Hofftag:benhoff.net,2019-12-10:/encouraging-open-source.html<p>Exploring methods for encouraging Open Source commits</p><p>I’ve been committing to OpenCV’s <span class="caps">CVAT</span>. This is for a couple of reasons. As I’ve gotten into more management and sales positions, I found that I didn’t have any reasons to keep my technical skills sharp. In fact, what I found with my last roles, is that by trying to commit to projects, I would often cause problems.</p>
<p>My coworkers were not thrilled to find out that my code introduced bugs. They also couldn’t count on me to be technically engaged, because the job pulled me in different directions. Me contributing to our tight turn project ended up being more of a distraction than a help, so I stopped.</p>
<p>So I set out to find a project that was related to the business space that we were in that I could work on while sharpening my technical skills. Specifically, at the time I was working for a custom services shop, and I was trying to push us to have a few computer vision <span class="caps">AI</span> projects. Thus we needed to have solid tools and experience to train the <span class="caps">AI</span> against. Bringing us back around to OpenCV’s <span class="caps">CVAT</span>.</p>
<p>Part of what’s interesting to me is building a community. So now that I’ve got a few commits into <span class="caps">CVAT</span>, including some buggy ones, it’s been interesting to me to see what works to help foster building a community.</p>
<p>I’ve already talked about <a href="https://benhoff.net/paying-for-open-source.html">trying to pay for open source contributions</a>. However I’ve yet to see any traction with that method.</p>
<p>One thing I do know, is that when people run into problems or have bugs, they are <span class="caps">HIGHLY</span> motivated to contribute. Sometimes challenging them to do so is enough to get them started (<a href="https://github.com/opencv/cvat/issues/501#issuecomment-503149106">it certainly worked on me</a>).</p>
<p>I’ve had some limited success with throwing out some high level ideas, linked to code to get successful code submissions. I think this is a decent practice depending on your familiarity with the code and the technical prowess of your consumers. Diagnosing and providing some ideas of how to fix stuff is often easy. It’s testing and iterating through the solution that becomes difficult.</p>
<p>So if you are running a similar project, that’s my suggestion. Challenge people, and if they run into a specific problem, throw them a couple of breadcrumbs.</p>
<p>You never know who might become a regular contributor to your project this way!</p>Integrating QCamera and QAbstractVideoSurface Together2019-12-10T20:28:00-06:002019-12-10T20:28:00-06:00Ben Hofftag:benhoff.net,2019-12-10:/qcamera-and-qabstractvideosurface.html<p>Working with a new client to integrate video and desktop together</p><p>I had a client approach me about doing a webcam integration user interface, which was great! I’ve been looking (<span class="caps">ITCHING</span>) for a reason to play with Qt’s video stack again.</p>
<p>Specifically the last time I left off with my explorations with Qt, I really wanted to integrate an Augmented Reality demo together with a webcamera. I wanted to integrate it with the Qt3D framework to show off the ability to integrate it all. I never could figure out how to do it (see <a href="https://stackoverflow.com/questions/50291828/how-to-display-an-image-in-qt3d">this Stackoverflow question</a>), but that doesn’t mean I had given up the problem.</p>
<p>Being technology greedy, I was also interested in learning how Qt’ QVideoFilter classes worked (see <a href="https://doc.qt.io/qt-5/qvideofilterrunnable.html">here</a>), to create a snapchat clone.</p>
<p>So you know, Augmented Reality, Snapchat, webcam-driven <span class="caps">QML</span> application. Just a casual Tuesday for me really.</p>
<p>However I ran into issues getting to the data (and getting Qt3D to display an image as noted above).</p>
<p>So when the client approached me to do an Qt and OpenCV integration, I knew I would finally have my chance to conquer Qt’s QCamera class.</p>
<p>Which is exactly what I did.</p>
<p>So the QCamera class has a <code>setViewfinder</code> method, where you can set an interface that can work with the raw data. In levels of concereteness to abstraction, the available classes are <code>QVideoWidget</code>, <code>QCameraViefinder</code> and <code>QAbstractVideoSurface</code>. QVideoWidget is a fully featured widget, whereas <code>QAbstractVideoSurface</code> is the abstract version ready to be subclassed and implemented.</p>
<p>When you dig through the source code for the more concrete classes, you get exposed to a lot of raw OpenGL. Having done some OpenGL programming before, I kind of know OpenGL, but I’m no expert. Additionally, it’s not anything I wanted to introduce into the class I was teaching at the time. So I kind of punted on the whole thing.</p>
<p>Creating a video course takes a lot of time, and I had to abandon any ideas that were too complex for my time frame. I couldn’t figure out if the OpenGL implementation made the program significantly more robust, because I couldn’t figure out if <span class="caps">NOT</span> implementing the program as OpenGL, would force me to download the data out of video memory and push it back up into video memory (a significant battery drainer for a mobile application). I don’t think that is the case, but I didn’t have time to explore. Luckily for this particular client, everything was on Desktop, so I didn’t need to worry about the specific battery constraints.</p>
<p>In order to implement QAbstractVideoSurface, you have to override/implement two methods: <code>supportedPixelFormats</code> and <code>present</code>. In the <code>supportedPixelFormats</code> method, you describe to the rest of the framework which types of QVideoFrame data formats that your implementation can support. My understanding is, if the data comes in a format that you don’t explicitly authorize, then the Qt framework uses the underlying system to convert the data into the format that you want.</p>
<p>Again, running on desktop and for a <span class="caps">MVP</span>, this wasn’t a big concern for me.</p>
<p>Instead, I wanted the system to handle as much of the conversion as possible, so I only exposed a single video format, <span class="caps">RGB32</span>.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">VideoSurface</span><span class="p">(</span><span class="n">QtMultimedia</span><span class="o">.</span><span class="n">QAbstractVideoSurface</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">supportedPixelFormats</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">handle_type</span><span class="p">):</span>
<span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">if</span> <span class="n">handle_type</span> <span class="o">==</span> <span class="n">QtMultimedia</span><span class="o">.</span><span class="n">QAbstractVideoBuffer</span><span class="o">.</span><span class="n">NoHandle</span><span class="p">:</span>
<span class="n">result</span> <span class="o">=</span> <span class="p">[</span><span class="n">QtMultimedia</span><span class="o">.</span><span class="n">QVideoFrame</span><span class="o">.</span><span class="n">Format_RGB32</span><span class="p">,]</span>
<span class="k">return</span> <span class="n">result</span>
</pre></div>
<p>This leaves only the <code>present</code> method. Present is where you actually handle the video data. I wanted to convert the video data into a format that I can use for the rest of my program.</p>
<p>I still find the image data types in Qt to be slightly confusing (I always have to look them up). Broadly though, I use primarily either QImage or QPixmap. I know QPixmap is to be used for display images, and I still wanted to manipulate my image, so I settled on QImage. So I had to convert my QVideoFrame to a QImage. Easy enough.</p>
<div class="highlight"><pre><span></span><span class="c1"># class VideoSurface(QtMultimedia.QAbstractVideoSurface):</span>
<span class="k">def</span> <span class="nf">present</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">frame</span><span class="p">:</span> <span class="n">QtMultimedia</span><span class="o">.</span><span class="n">QVideoFrame</span><span class="p">):</span>
<span class="n">copy</span> <span class="o">=</span> <span class="n">QtMultimedia</span><span class="o">.</span><span class="n">QVideoFrame</span><span class="p">(</span><span class="n">frame</span><span class="p">)</span>
<span class="n">copy</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">QtMultimedia</span><span class="o">.</span><span class="n">QAbstractVideoBuffer</span><span class="o">.</span><span class="n">ReadOnly</span><span class="p">)</span>
<span class="n">image_format</span> <span class="o">=</span> <span class="n">QtMultimedia</span><span class="o">.</span><span class="n">QVideoFrame</span><span class="o">.</span><span class="n">imageFormatFromPixelFormat</span><span class="p">(</span><span class="n">copy</span><span class="o">.</span><span class="n">pixelFormat</span><span class="p">())</span>
<span class="n">my_image</span> <span class="o">=</span> <span class="n">QtGui</span><span class="o">.</span><span class="n">QImage</span><span class="p">(</span><span class="n">copy</span><span class="o">.</span><span class="n">bits</span><span class="p">(),</span> <span class="n">copy</span><span class="o">.</span><span class="n">width</span><span class="p">(),</span> <span class="n">copy</span><span class="o">.</span><span class="n">height</span><span class="p">(),</span> <span class="n">copy</span><span class="o">.</span><span class="n">bytesPerLine</span><span class="p">(),</span> <span class="n">QtGui</span><span class="o">.</span><span class="n">QImage</span><span class="o">.</span><span class="n">Format</span><span class="p">(</span><span class="n">image_format</span><span class="p">))</span>
<span class="n">copy</span><span class="o">.</span><span class="n">unmap</span><span class="p">()</span>
<span class="c1"># Do something with your new `QImage` here!</span>
<span class="k">return</span> <span class="bp">True</span>
</pre></div>
<p>Now I finally had what I wanted all that time ago! A simple example of low level access out of a QCamera. The only thing to do is wire it up to a QCamera</p>
<div class="highlight"><pre><span></span><span class="n">camera</span> <span class="o">=</span> <span class="n">QtMultimedia</span><span class="o">.</span><span class="n">QCamera</span><span class="p">()</span>
<span class="n">video_surface</span> <span class="o">=</span> <span class="n">VideoSurface</span><span class="p">()</span>
<span class="n">camera</span><span class="o">.</span><span class="n">unload</span><span class="p">()</span>
<span class="n">camera</span><span class="o">.</span><span class="n">setViewfinder</span><span class="p">(</span><span class="n">video_surface</span><span class="p">)</span>
<span class="n">camera</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
</pre></div>Paying for Open Source2019-11-26T06:40:00-06:002019-11-26T06:40:00-06:00Ben Hofftag:benhoff.net,2019-11-26:/paying-for-open-source.html<p>Exploring methods for promoting Open Source</p><p>I think the first time I saw an Open Source project that was well funded and was using that funding to promote development was NeoVim. NeoVim was a fork of the popular Vim project, focused on reducing the cruft of the project and bringing best practices back in. Whoever was running the project was a marketing genius. He got a ton of press outlining the issues with the Vim project. But the key difference between him and a thousand of other people complaining about projects is that he launched an alternative <em>and asked for money</em>. This is how I got introduced to <a href="https://www.bountysource.com/">BountySource</a>.</p>
<p>The BountySource idea is simple. Give money against (typically) GitHub issues to promote development for fixes. Bounties can be pooled so that interested individuals can pool money together. Some of the bounties are quite large, in the $5K realm and <span class="caps">IBM</span> has awarded $470,000+ dollars on the platform. Now these two data points are by far the outliers. The platform itself hasn’t really caught on in a substantial way. There are 5-8 projects in the $5K realm which are deeply technical lifts, and the next largest businesses are Ripple coin and Elementary with $37K and $27K respectively.</p>
<p>NeoVim itself looks like it’s focused on funding an individual developer or project maintenance over funding individual issues. I seem to remember in the early days that they were funding individual issues. I’m guessing the lag between putting money down and actual implementations was too large. Especially for bug fixes, this seems like an unwise strategy.</p>
<p>For my own personal use, I’ve been interested in throwing some small cash sums around to get a feel for different approaches. My pet project lately has been <a href="https://benhoff.net/developing-cvat.html">OpenCV’s <span class="caps">CVAT</span></a>. Because <span class="caps">CVAT</span> has been blending in Machine Learning into the user interface, it’s a cool sell. You can show people the platform itself and the neat things it can do.</p>
<p>I wanted to continue to increase the magicalness of the platform, so I put <a href="https://www.bountysource.com/issues/70724425-intelligent-scissors">$100 on BountySource</a> against some <a href="https://github.com/opencv/cvat/issues/336">intelligent scissors</a>.</p>
<p>Nobody has bitten off on the project. I’m guessing there’s a couple of issues, one of which is that the dollar value is low for the platform. The other of which is I haven’t done any marketing. BountySource doesn’t seem like a large enough of a platform to drive implementations for that low of a price point. And at least for United States development labor, the cost is probably a bit low. I would guess that patch would take somewhere between 4-32 hours to implement, depending on the developer. That’s probably not costing time for setting up a development environment with a new project. Assuming $50/hour, it’s quick to see the price delta.</p>Visual Debugging from the Command Line2019-11-24T22:39:00-06:002019-11-24T22:39:00-06:00Ben Hofftag:benhoff.net,2019-11-24:/command-line-visual-debugging.html<p>How to visually debug something from the command line</p><p>Working with images is interesting. In a previous piece of work, a customer wanted to bring in some semantic segmentation. Semantic Segmentation goes pixel by pixel and assigns probabilities that the pixel belongs to label. You’ll typically see this in street images, with individual cars, people, bikes, etc being segmented out of the overall picture on a pixel level.</p>
<p>The interesting problem of working with semantic segmentation technologies is working with the outputs. Typically the output of a semantic segmentation model is a list of pixel probabilities. Because it’s a pixel-by-pixel technology, you end up with probabilities for every single pixel in an image. Combine that with an estimation for each type of class that a pixel can be, and every pixel in an 800 x 600 image suddenly has 50 probabilities associated with it.</p>
<p>(There’s some tech where they use the bounding box to limit the number of pixels passed back (Mask <span class="caps">RCNN</span>), but it’s still a lot of data to work with.)</p>
<p>I’m not sure I’ve ever figured out a good way to work with multi-layered array data like that contained within pictures. Leave a comment below if you have any hints.</p>
<p>Since I was working with images though, the nice thing is that you can just, you know, <em>display</em> the images. After much trail and error, that was the best way I could figure out how to actually iteratively develop semantic segmentation code. </p>
<p>Interestingly, I’ve never much cared for how OpenCv handles windows until know.</p>
<div class="highlight"><pre><span></span><span class="c1"># Note that `some_numpy_array` is just that, a numpy array</span>
<span class="c1"># Visualize the array or image</span>
<span class="n">cv2</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="s1">'My Array'</span><span class="p">,</span> <span class="n">some_numpy_array</span><span class="p">)</span>
<span class="c1"># wait until user presses keys</span>
<span class="c1"># Note that without this waitkeys, the window will be created and destroyed before you can notice it</span>
<span class="n">cv2</span><span class="o">.</span><span class="n">waitKeys</span><span class="p">()</span>
<span class="c1"># Do some kind of manipulation here or you could show another picture to compare</span>
<span class="n">some_numpy_array</span> <span class="o">=</span> <span class="n">do_some_action</span><span class="p">(</span><span class="n">some_numpy_array</span><span class="p">)</span>
<span class="c1"># Visualize the array or image again after the changes</span>
<span class="n">cv2</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="s1">'My Array'</span><span class="p">,</span> <span class="n">some_numpy_array</span><span class="p">)</span>
<span class="n">cv2</span><span class="o">.</span><span class="n">waitKeys</span><span class="p">()</span>
</pre></div>
<p>The method cv2.destroyAllWindows() or cv2.destroyWindow(‘your-name-here’) might be useful depending on the specifics of what you have to visualize. I just reused the existing context/window, but there might be cases where you’d want to display multiple windows side-by-side or some other setup.</p>
<p>This was a funny setup because I was actually building some server code. So as I was debugging, my web server started spitting some Desktop Graphical User Interfaces. While none of this code made it into production, it definitely helped me track down the logic errors in my code and get the result <a href="https://github.com/opencv/cvat/commit/a435b410ede92130ca2bc67a7ae2a60962f6ebef">submitted up and working correctly</a>.</p>Developing OpenCV’s CVAT2019-11-24T21:39:00-06:002019-11-24T21:39:00-06:00Ben Hofftag:benhoff.net,2019-11-24:/developing-cvat.html<p>Summarizing my work up to date with <span class="caps">CVAT</span></p><p>As part of a work project, I needed to showcase off some computer vision (specifically, semantic segmentation) capabilities. The client needed to be able to understand both what was technically possibly, but also feel comfortable that it was mature enough to be put into production. I’ve found that finding open source projects that are being driven by big name players typically gives the required comfort to customers that the tech is mature. Additionally, having a sweet looking and usable user interface that you can drop in to the project day 1 is also very helpful.</p>
<p>Enter <span class="caps">CVAT</span>. Intel has recently been trying to flex it’s muscle in the deep learning space, making up for the early lead that <span class="caps">NVIDIA</span> has taken with <span class="caps">CUDA</span>. Intel’s approach has been an open source one, leading with a suite of deep learning technologies. I’m a little ignorant on the total depth of all the total tools, but I am now very familiar with <span class="caps">CVAT</span> and OpenVINO tool suites.</p>
<p><span class="caps">CVAT</span> is an image annotation platform to be used for training machine learning models. While most image annotation labor is cheap (literally drawing boxes on a static image), there’s always a desire to leverage existing work to increase the accuracy, speed, and reduce costs in annotation labor.</p>
<p>To do this, <span class="caps">CVAT</span> integrates open source pretrained models to act as “first-pass” annotators using the OpenVINO toolkit.</p>
<p>OpenVINO is an abstracting layer between multiple toolkits, allowing one to use both PyTorch, <span class="caps">ONNX</span>, and Tensorflow models in a uniform way. (There’s also some optimizations it uses to abstract between <span class="caps">CPU</span> and <span class="caps">GPU</span> power, but that’s less important for this use case).</p>
<p><span class="caps">CVAT</span> was perfect for my needs. Someone else had already built the user interface, I just needed to customize a bit of the OpenVINO code base to showcase what was possible for my client. Well, that was the theory.</p>
<p>In reality, the integration between the web server and the deep learning hadn’t had much main stream development support. So while what I wanted to do was theoretically easy, it actually took significant development support to make it possible.</p>
<p>In addition to a few bugs that had to be fixed, the biggest technical contributions I had to do, were quite odd. Users were required to submit Python scripts to process the results of each model back into a format that <span class="caps">CVAT</span> understood. The architects of the code that I was using had set up a limited python sandbox to process this user submitted code. When I first started, there was no way for a user to get the full power of the python language , even if you were an admin. This was obviously not going to work for me. So the first thing I did was fix that. <a href="https://github.com/opencv/cvat/commit/f20698921e5355a4b13c4f90612163f56b9835d0">I added a flag to check if a user was a admin or not and gave them the capability to run all of python if they were an admin</a>. That fixed the majority of my issues, but I still didn’t have the full power of Python behind me. Because Python’s eval does not actually process import statements.</p>
<p>So the second thing I had to do was add in the capability to import code in these user-submitted scripts. In order to do that, I would have to programmatically assess and import the required packages before running the “eval” statement.</p>
<p>I originally thought that setting up this process was going to be too difficult. However, I did remember that Python has actually exposed a significant amount of it’s code processing in the standard library, and that gave me hope. With the help of a <a href="https://stackoverflow.com/questions/9008451/python-easy-way-to-read-all-import-statements-from-py-module">StackOverflow question</a>, I found out that I could indeed quickly parse out the import statements using Python’s “ast” library. The only thing left to do was to <a href="https://github.com/opencv/cvat/commit/418cdbe1464a06d5d7b26e6385586a2e5cfa12a5">parse the import statements and dynamically import the code, before evaluating the code</a>.</p>
<p>The most technically impressive peace of this actually ended up being the easiest due to the above linked StackOverflow question (minus the research to get there). Tracing through the code base to manually ensure admin checks were put in place (my first patch) took significantly to ensure everything was in correctly. Grokking code completely and dealing with permissions is difficult, and will always probably be.</p>
<p>However, with the capability to showcase off semantic segmentation in a fully featured web interface, my customer was convinced of the readiness of the technology. Sadly, I left the company before I was able to get the contract in place. I did have a lot of fun building out the tech!</p>Rescuing the Blog2019-11-22T21:39:00-06:002019-11-22T21:39:00-06:00Ben Hofftag:benhoff.net,2019-11-22:/rescuing-the-blog.html<p>Getting a project running that I haven’t touched in awhile</p><p>I haven’t blogged in awhile! While I’ve been keeping up the discourse that’s attached to the site, the actual blog itself is a different story. The static nature of the site helps keep it maintained, because I don’t actually have to touch it, until I want to add new content.</p>
<p>One thing that I noticed right away is that I’ve switched to using Markdown over Restructed text. This is because Github uses Markdown. So this article is being written using Markdown instead of <span class="caps">RST</span>.</p>
<p>The other thing I’ve learned is that I need to keep idiot proof instructions for how to stand this thing back up after I’ve been away for awhile. It’s funny how when you have side projects, the first thing that you notice is that you need a “look at this stupid!” text blob for remembering how to get things running again.</p>
<p>My instructions for this project are awful. I’ve added some updates to remind myself. I was surprised on a fresh install that all the dependencies seem to be working after two years! It seems that chooosing Pelican was a smart choice.</p>
<p>Now let’s see if we can’t push these two articles up and close the loop.</p>
<p>If I remember the most difficult part with the entire process, it’s actually in the pushing of the content.</p>Python Extra Requires Woes2019-11-21T21:39:00-06:002019-11-21T21:39:00-06:00Ben Hofftag:benhoff.net,2019-11-21:/python-extras-requires.html<p>Installing extras when running setup.py</p><p>One of the small things that I always forget or abuse in projects is “extras_requires”. In python project “setup.py” file, you can drop small hints, or tier out the installation requirements so that users don’t have to download every single dependency in the world to get your project running. This is achieved by the use of “extras_requires”.</p>
<p>The problem is, I never remember how to install the extras when using my desired installation method of:</p>
<div class="highlight"><pre><span></span>$ python setup.py develop<span class="sb">`</span>
</pre></div>
<p>Currently the only way I know how to do it is to do a “pip install -e .[my-extras-here]”. I would like to be able to define the extras requires during development install. I’ve opened a <a href="https://stackoverflow.com/questions/58986867/use-python-setup-py-to-install-different-dependencies-with-develop-vs-install">Stackoverflow question</a> to find out if it’s possible. But based on some research, I don’t think there is any way to do it.</p>
<p>I’ll just change the project for now to include all the dependencies for my desired use case. I think the use of “extras_requires” for most of my personal projects is probably an anti-pattern.</p>Using Git Branches With Setuptools2017-12-28T00:00:00-06:002017-12-28T00:00:00-06:00Ben Hofftag:benhoff.net,2017-12-28:/using-git-branches-with-setuptools.html<p class="first last">Want to specify a git dependency in a setup.py? Here’s how.</p>
<p>I wanted to add a git branch in a dependency for a project of mine in the <cite>setup.py</cite> file. Should be easy right?</p>
<p>Several hours of research later and pinging a project maintainer to bump the version in his branch later…</p>
<p>And it’s actual not too bad once you know how to do it. So let’s figure out how to do it.</p>
<p>In the <cite>setup</cite> method of <cite>setup.py</cite> you need to have two arguments.</p>
<ul class="simple">
<li>install_requires</li>
<li>dependency_links</li>
</ul>
<p>In this case, I was trying to add the <a class="reference external" href="https://github.com/jonathanslenders/python-prompt-toolkit">python-prompt-toolkit</a> 2.0.0 branch to my project (Version 1.0.15 is currently in pypi).</p>
<p>Let’s pick the easy one first, the install_requires.</p>
<div class="highlight"><pre><span></span><span class="n">install_requires</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'prompt-toolkit>=2.0.0'</span><span class="p">]</span>
</pre></div>
<p>This requirement parsing will currently fail due to the fact there isn’t a prompt-toolkit version greater than or equal to 2.0.0 in pypi. So let’s fix that by specifying the 2.0.0 branch with in the dependency links.</p>
<div class="highlight"><pre><span></span><span class="n">dependency_links</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'git+https://github.com/jonathanslenders/python-prompt-toolkit@2.0#egg=prompt-toolkit-2.0.0'</span><span class="p">]</span>
</pre></div>
<p>There’s a couple of special things about the way this link is written that are poorly documented, and thus worth writing about.</p>
<ol class="arabic simple">
<li><cite>git+https://github.com/jonathanslenders/python-prompt-toolkit</cite> -> Standard documented way to install git links using pip. Nothing to see here, just an <span class="caps">FYI</span>.</li>
<li>The <cite>@2.0</cite> specifies the branch that we want to use. Note that this happens to be a number in this case, but can be whatever branch you need. I.e., <cite>dev</cite> would be a common interesting branch name that could be used by specifying <cite>@dev</cite>.</li>
<li>The <cite>#egg=prompt-toolkit-2.0.0</cite> is the most important bit and worth expounding on a little more.</li>
</ol>
<p>We need to specify which package this dependency link provides. We do this with the <cite>#egg=PACKAGE_NAME</cite> syntax. In this case, this dependency link provides the package <cite>prompt-toolkit</cite>, ie <cite>#egg=prompt-toolkit</cite></p>
<p>But the most important part of this portion of a dependency link, and one that isn’t documented at all, is that you <em>must</em> provide a version number. I’ve done this here with the <cite>-2.0.0</cite>. The version number you provide can be any version, but it must be there.</p>
<p>So for example, if you wanted to specify a dependency on the development version of Vexbot, the dependency link could look like this:</p>
<div class="highlight"><pre><span></span><span class="n">dependency_links</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'git+https://github.com/benhoff/vexbot@dev#egg=vexbot-0'</span><span class="p">]</span>
</pre></div>
<p>Note the <cite>-0</cite> specifying the version number at the end of the vexbot dependency link.</p>
<p>You can throw in whatever version number you’d like so that you can help properly guide your user as to what version you really need (like I did with the <cite>install_requires</cite> constraint on prompt-toolkit).</p>
<p>Please note that dependency links are useful! You can use them to:</p>
<ul class="simple">
<li>Specify a private git repository as a dependency in setup.py</li>
<li>Specify a patch version of a repository over the package in pypi</li>
<li>Specify a package that isn’t published to pypi but is on Github</li>
</ul>
<p>Hope that helps!</p>
Self Hosted Python Anywhere2017-10-16T07:22:00-05:002017-10-16T07:22:00-05:00Ben Hofftag:benhoff.net,2017-10-16:/local-python-anywhere.html<p class="first last">Working around aggressive workplace firewalls</p>
<p>I have a great job, but it doesn’t always fully employ me. Such is life. I still like to be productive, and having a programming hobbies and an office job seems like a match made in heaven.</p>
<p>Oh, you don’t have enough things for me to do? But you still want me to be here in case something comes up? No worries team, I gotcha. Totally self entertaining.</p>
<p>The problem has been that my office’s aggressive firewall has prevented me from doing anything that doesn’t come over port 80 or 443. Which I totally get. But c’mon man. I can only read Reddit so many hours of the day.</p>
<p>But then I found <a class="reference external" href="https://www.pythonanywhere.com/">Python Anywhere</a>. The basic premise is that all you need is a web browser and you can be up and coding in no time. I immediately signed up. This was exactly what I was looking for! Up until the point I realized that I can’t bind to ports. With my main project using zmq, which requires binding to ports, I was super bummed.</p>
<p>It did however get me thinking that maybe I could use the same program. They had a full fledged terminal in the browser! There was no way that the website author made that program himself. It was too slick. It had to be open sourced.</p>
<p>After lots of googling I found it. <a class="reference external" href="https://github.com/macton/hterm">hterm</a>, the xterm-compatible terminal emulator. A terminal emulator itself didn’t help me though. I need to have it hosted and hooked into a server somewhere. Enter <a class="reference external" href="https://github.com/krishnasrinivas/wetty">wetty</a>, the full bound service.</p>
<p>So now that I had a terminal as a service, I still needed to work with my office’s firewall. Luckily for me, I have this blog. Or more specifically, I have the comments of the blog that I’m hosting myself using nginx. Using a <a class="reference external" href="https://en.wikipedia.org/wiki/Reverse_proxy">reverse proxy</a> feature in nginx, I was able to link up my new ‘terminal as a service’ to the ip address of my blog comments which has already been white listed in my organizations firewall.</p>
<p>Sweet. My organization gets a much more motivated/attentive employee during lull periods and I get to be more productive. Win-win.</p>
Dealing with Deeply Nested Data Structures2017-10-05T10:24:00-05:002017-10-05T10:24:00-05:00Ben Hofftag:benhoff.net,2017-10-05:/interactive-debugging-methods.html<p class="first last">Showcasing a neat way to explore deeply nested data structures</p>
<p>Working with deeply nested data structures is a pain. Sometimes the easiest way to figure out the control structure to get the data that you want is to play around with the data the <a class="reference external" href="https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop"><span class="caps">REPL</span></a>. My problem is comes from an <span class="caps">API</span>, it’s a pain to figure out how to quickly evaluate the structure, make lasting changes to the script, and then jump back into figuring out the structure. In the past, I’ve sucked it up and used the <span class="caps">REPL</span>, I’ve written scripts and eval’d integrated the data one change at a time using <cite>print</cite>, and occasionally, I’ll use jupyter-notebook. None of these solutions seemed to be very good though.</p>
<p>I still haven’t figured out a great way, but yesterday when working with json data from Reddit, the thought occurred to me that I might be able to jump straight into the <span class="caps">REPL</span> and just pass in the scope that I was working in. I’d seen this done in the <a class="reference external" href="https://github.com/gawel/irc3">irc3</a> code, but I couldn’t figure out how that would be useful at the high level that it was done at (basically interrogating the final <cite>bot</cite> instance).</p>
<p>Sure enough, you can definitely do it. Using the <cite>interact</cite> method from the <cite>code</cite> module.</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">code</span> <span class="kn">import</span> <span class="n">interact</span>
<span class="c1"># some expensive calls here to get....</span>
<span class="c1"># ...a deeply nested data structures</span>
<span class="n">data</span> <span class="o">=</span> <span class="p">[[{},</span> <span class="p">{},</span> <span class="p">{}],</span> <span class="p">[{},</span> <span class="p">{}],</span> <span class="p">[{},</span> <span class="p">{},</span> <span class="p">{},</span> <span class="p">{}]]</span>
<span class="k">for</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">data</span><span class="p">:</span>
<span class="k">for</span> <span class="n">deeper_value</span> <span class="ow">in</span> <span class="n">value</span><span class="p">:</span>
<span class="n">interact</span><span class="p">(</span><span class="n">local</span><span class="o">=</span><span class="nb">locals</span><span class="p">())</span>
</pre></div>
<p>Dropping into the <span class="caps">REPL</span> like this gives me the ability to query <cite>deeper_value</cite> interactively. This is often useful when dealing with <span class="caps">JSON</span> to figure out what keys are available.</p>
Creating a Bot that Quotes Pop Culture in Context2017-10-04T06:15:00-05:002017-10-04T06:15:00-05:00Ben Hofftag:benhoff.net,2017-10-04:/movie-quotes-in-context.html<p class="first last">Working with Reddit comments and machine learning to get movie quote context</p>
<p>So I’m interested in creating a bot that could quote movie quotes back to your in a context that makes sense. I don’t know how your conversations with your buddies go, but 70% of our everyday conversation is quoting things in new and different contexts, so I was interested to see if I could collect enough data to do this with some common quotes.</p>
<p>The first part of any good machine learning project is to collect data. I read <a class="reference external" href="http://www.zmescience.com/science/reddit-supercomp-59815/">that OpenAI was using Reddit comments</a> to learn language, so I figured I’d use Reddit as a source. Live data seemed to be the most interesting way to pull data out.</p>
<p>The problem is that I’ve vastly overestimated (apparently) how much people quote famous movie quotes. I left the program running overnight and instead of having 100’s of matches, I only had one. So I’ll have to figure out a different way to get data out.</p>
<p>From a technology standpoint, this was a relatively easy problem to solve. I’m using <a class="reference external" href="https://praw.readthedocs.io/en/latest/"><span class="caps">PRAW</span></a>, the Python Reddit <span class="caps">API</span> wrapper to get comments out. I wanted some flexibility for spelling, so I ended up using the <a class="reference external" href="https://en.wikipedia.org/wiki/Levenshtein_distance">Levenshtein algorithim</a> from the <a class="reference external" href="https://github.com/ztane/python-Levenshtein/">python-Levenshtien</a> package. There’s probably a better algo to do this, see <a class="reference external" href="https://stackoverflow.com/questions/3338889/how-to-find-similar-results-and-sort-by-similarity">this stackoverflow post</a> about that issue, but Levenshtien was good enough for a proof of concept.</p>
<p>The plan was to grab all of the parent comment’s text if there was a match that was good close to any of the my movie quotes. But like I said, there’s not enough data being returned from grabbing live comments to make this feasible.</p>
<p>I ran this in tmux on a digital ocean instance I have running for 12 hours. You could also have this write to a file as well. I turned off the similarity matching and have just been watching the raw text come in while I write this post.</p>
<p>Man, that ‘Remind me!’ bot is popular.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">praw</span>
<span class="kn">from</span> <span class="nn">Levenshtein</span> <span class="kn">import</span> <span class="n">ratio</span>
<span class="n">config</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'client_id'</span><span class="p">:</span> <span class="s1">''</span>
<span class="s1">'client_secret'</span><span class="p">:</span> <span class="s1">''</span><span class="p">,</span>
<span class="s1">'password'</span><span class="p">:</span> <span class="s1">''</span><span class="p">,</span>
<span class="s1">'username'</span><span class="p">:</span> <span class="s1">''</span><span class="p">}</span>
<span class="c1"># quotes is a list of string</span>
<span class="n">quotes</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">USER_AGENT</span> <span class="o">=</span> <span class="s1">'Movie Quote Bot by /u/beohoff'</span>
<span class="n">reddit</span> <span class="o">=</span> <span class="n">praw</span><span class="o">.</span><span class="n">Reddit</span><span class="p">(</span><span class="n">client_id</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s1">'client_id'</span><span class="p">],</span>
<span class="n">client_secret</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s1">'client_secret'</span><span class="p">],</span>
<span class="n">password</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s1">'password'</span><span class="p">],</span>
<span class="n">username</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s1">'username'</span><span class="p">],</span>
<span class="n">user_agent</span><span class="o">=</span><span class="n">USER_AGENT</span><span class="p">)</span>
<span class="k">for</span> <span class="n">quote</span> <span class="ow">in</span> <span class="n">quotes</span><span class="p">:</span>
<span class="n">quote_len</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">quote</span><span class="p">)</span>
<span class="k">if</span> <span class="n">quote_len</span> <span class="o">></span> <span class="n">greatest_length</span><span class="p">:</span>
<span class="n">greatest_length</span> <span class="o">=</span> <span class="n">quote_len</span>
<span class="k">if</span> <span class="n">quote_len</span> <span class="o"><</span> <span class="n">least_length</span><span class="p">:</span>
<span class="n">least_length</span> <span class="o">=</span> <span class="n">quote_len</span>
<span class="k">for</span> <span class="n">comment</span> <span class="ow">in</span> <span class="n">reddit</span><span class="o">.</span><span class="n">subreddit</span><span class="p">(</span><span class="s1">'AskReddit+movies+funny+pics'</span><span class="p">)</span><span class="o">.</span><span class="n">stream</span><span class="o">.</span><span class="n">comments</span><span class="p">():</span>
<span class="n">text</span> <span class="o">=</span> <span class="n">comment</span><span class="o">.</span><span class="n">body</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span>
<span class="n">len_text</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
<span class="k">if</span> <span class="n">len_text</span> <span class="o">+</span> <span class="mi">7</span> <span class="o">></span> <span class="n">greatest_length</span> <span class="ow">or</span> <span class="n">len_text</span> <span class="o">-</span> <span class="mi">4</span> <span class="o"><</span> <span class="n">least_length</span><span class="p">:</span>
<span class="k">continue</span>
<span class="n">greatest</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">best_quote</span> <span class="o">=</span> <span class="s1">''</span>
<span class="k">for</span> <span class="n">quote</span> <span class="ow">in</span> <span class="n">quotes</span><span class="p">:</span>
<span class="n">value</span> <span class="o">=</span> <span class="n">ratio</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">quote</span><span class="p">)</span>
<span class="k">if</span> <span class="n">value</span> <span class="o">></span> <span class="o">.</span><span class="mi">75</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">quote</span><span class="p">)</span>
</pre></div>
<p>Guess it’s back to the drawing board for how to get enough data to create a bot that can respond in context with movie quotes.</p>
PyQt Signals and Slots2017-07-21T07:40:00-05:002017-07-21T07:40:00-05:00Ben Hofftag:benhoff.net,2017-07-21:/pyqt-signals-slots.html<p class="first last">Figuring out how to use signals and slots</p>
<p>As part of our <a class="reference external" href="https://benhoff.net/pyqt-tutorial.rst">PyQt Tutorial series</a>, we’ve gone through some basic <a class="reference external" href="https://benhoff.net/pyqt-layout-design.rst">layout management</a> in addition to a conversation about some <a class="reference external" href="https://benhoff.net/qt-interface-design.rst">interface design</a>… but now when I click buttons I want things to happen!</p>
<p>In order to achieve that goal, we’re going to have to learn about signals and slots.</p>
<p>Let me let you in on a little secret. Signals and slots? They’re magical. Seriously, they are pretty cool.</p>
<p>Let’s go back to our face recognition example. If you’re jumping around, you can catch up to the source code that we’re starting at <a class="reference external" href="https://github.com/benhoff/blog/blob/master/scripts/pyqt-layout-design.py">here</a>. This time, since we know layouts due to the <a class="reference external" href="https://benhoff.net/pyqt-layout-design.rst">layout management</a> post, we’re going to build our own widget so that we can better hook up our signals and slots.</p>
<p>This is going to track closely to the <a class="reference external" href="https://benhoff.net/face-detection-in-pyqt.rst">face detection post</a> where I originally created this widget.</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">PyQt5</span> <span class="kn">import</span> <span class="n">QtCore</span><span class="p">,</span> <span class="n">QtWidgets</span>
<span class="kn">from</span> <span class="nn">facerecog</span> <span class="kn">import</span> <span class="p">(</span><span class="n">RecordVideo</span><span class="p">,</span>
<span class="n">FaceDetectionWidget</span><span class="p">,</span>
<span class="n">get_haarcascade_filepath</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">FaceRecogControl</span><span class="p">(</span><span class="n">QtWidgets</span><span class="o">.</span><span class="n">QWidget</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">parent</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">parent</span><span class="p">)</span>
<span class="n">haar_filepath</span> <span class="o">=</span> <span class="n">get_haarcascade_filepath</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">face_detect_widget</span> <span class="o">=</span> <span class="n">FaceDetectionWidget</span><span class="p">(</span><span class="n">haar_filepath</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">video_recording</span> <span class="o">=</span> <span class="n">RecordVideo</span><span class="p">()</span>
<span class="n">record_button</span> <span class="o">=</span> <span class="n">QtWidgets</span><span class="o">.</span><span class="n">QPushButton</span><span class="p">(</span><span class="s1">'Run'</span><span class="p">)</span>
<span class="n">layout</span> <span class="o">=</span> <span class="n">Qtwidgets</span><span class="o">.</span><span class="n">QVBoxLayout</span><span class="p">()</span>
<span class="n">layout</span><span class="o">.</span><span class="n">addWidget</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">face_detect_widget</span><span class="p">)</span>
<span class="n">layout</span><span class="o">.</span><span class="n">addWidget</span><span class="p">(</span><span class="n">record_button</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">setLayout</span><span class="p">(</span><span class="n">layout</span><span class="p">)</span>
</pre></div>
<p>You’ll notice that in the code above, I didn’t put the <tt class="docutils literal">QPushButton</tt> (instance member name of <tt class="docutils literal">record_button</tt>), as a instance member. Since I added the push button to our layout, the layout will actually keep a reference to the instance, preventing garbage collection.</p>
<p>So all of that code should be review. Create a layout, add some widgets to the layout, and then set the layout on our widget.</p>
<p>Now let’s go ahead and wire our creation up using signals and slots.</p>
<p>As the <a class="reference external" href="http://doc.qt.io/qt-5/signalsandslots.html">documentation states</a>, signals and slots are used for communication between objects. In this case, we want to communicate between our push button object and our record video object. Specially, when we push the “Run” button, we want our video recording object to start recording.</p>
<p>So looking at the <a class="reference external" href="http://doc.qt.io/qt-5/qabstractbutton.html#signals">push button documentation</a>, we can see that we have several signals available to us. The one we’re interested in is <tt class="docutils literal">clicked</tt>. Now the function that we want called after our button is <tt class="docutils literal">clicked</tt> is the <tt class="docutils literal">start_recording</tt> method on the <tt class="docutils literal">VideoRecord</tt> instance. To do this, we’ll call the <tt class="docutils literal">connect</tt> method on the <tt class="docutils literal">clicked</tt> class instance and pass our <tt class="docutils literal">start_recording</tt> method in as the argument. We also need to wire our <tt class="docutils literal">image_data</tt> signal to our <tt class="docutils literal">image_data_slot</tt>. That’s a lot of words. Let’s see it in action.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">FaceRecogControl</span><span class="p">(</span><span class="n">QtWidgets</span><span class="o">.</span><span class="n">QWidget</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">parent</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">parent</span><span class="p">)</span>
<span class="n">haar_filepath</span> <span class="o">=</span> <span class="n">get_haarcascade_filepath</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">face_detect_widget</span> <span class="o">=</span> <span class="n">FaceDetectionWidget</span><span class="p">(</span><span class="n">haar_filepath</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">video_recording</span> <span class="o">=</span> <span class="n">RecordVideo</span><span class="p">()</span>
<span class="n">record_button</span> <span class="o">=</span> <span class="n">QtWidgets</span><span class="o">.</span><span class="n">QPushButton</span><span class="p">(</span><span class="s1">'Run'</span><span class="p">)</span>
<span class="n">layout</span> <span class="o">=</span> <span class="n">Qtwidgets</span><span class="o">.</span><span class="n">QVBoxLayout</span><span class="p">()</span>
<span class="n">layout</span><span class="o">.</span><span class="n">addWidget</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">face_detect_widget</span><span class="p">)</span>
<span class="n">layout</span><span class="o">.</span><span class="n">addWidget</span><span class="p">(</span><span class="n">record_button</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">setLayout</span><span class="p">(</span><span class="n">layout</span><span class="p">)</span>
<span class="c1"># Connect our signal `clicked` to our method `start_recording`</span>
<span class="n">record_button</span><span class="o">.</span><span class="n">clicked</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">video_recording</span><span class="o">.</span><span class="n">start_recording</span><span class="p">)</span>
<span class="c1"># alias out the method call `image_data_slot` to make the code</span>
<span class="c1"># line shorter</span>
<span class="n">image_data_slot</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">face_detect_widget</span><span class="o">.</span><span class="n">image_data_slot</span>
<span class="c1"># connect our signal `image_data` to our method `image_data_slot`</span>
<span class="bp">self</span><span class="o">.</span><span class="n">video_recording</span><span class="o">.</span><span class="n">image_data</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">image_data_slot</span><span class="p">)</span>
</pre></div>
<p>In PyQt, we can connect signals to any method call as long as the signatures match. In the case of our <tt class="docutils literal">clicked</tt> method, no arguments are transmitted when the signal is emitted. However, if we look at the <a class="reference external" href="http://doc.qt.io/qt-5/qcombobox.html">QComboBox signal documentation</a>, we’ll see that some of the signals (<tt class="docutils literal">activated</tt> for example) emit arguments that we need to catch in our method call.</p>
<p>Let’s go ahead and define our own custom signal. For example, maybe we want to transmit a signal whenever a face is detected in our widget. Let’s go ahead and subclass our <tt class="docutils literal">FaceDetectionWidget</tt>. We’ll create a <tt class="docutils literal">face_detected</tt> signal and override our <tt class="docutils literal">image_data_slot</tt> method to emit the face detected signal whenever we find a face.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">FaceSignalWidget</span><span class="p">(</span><span class="n">FaceDetectionWidget</span><span class="p">):</span>
<span class="c1"># Create our signal</span>
<span class="n">face_detected</span> <span class="o">=</span> <span class="n">QtCore</span><span class="o">.</span><span class="n">pyqtSignal</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">image_data_slot</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">image_data</span><span class="p">):</span>
<span class="n">faces</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">detect_faces</span><span class="p">(</span><span class="n">image_data</span><span class="p">)</span>
<span class="c1"># If faces our found, `emit` our signal</span>
<span class="k">if</span> <span class="n">faces</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">face_detected</span><span class="o">.</span><span class="n">emit</span><span class="p">()</span>
<span class="c1"># NOTE: this code is same as base class ----------------------------</span>
<span class="k">for</span> <span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">h</span><span class="p">)</span> <span class="ow">in</span> <span class="n">faces</span><span class="p">:</span>
<span class="n">cv2</span><span class="o">.</span><span class="n">rectangle</span><span class="p">(</span><span class="n">image_data</span><span class="p">,</span> <span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">),</span> <span class="p">(</span><span class="n">x</span><span class="o">+</span><span class="n">w</span><span class="p">,</span> <span class="n">y</span><span class="o">+</span><span class="n">h</span><span class="p">),</span> <span class="bp">self</span><span class="o">.</span><span class="n">_red</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_width</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">image</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_qimage</span><span class="p">(</span><span class="n">image_data</span><span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">image</span><span class="o">.</span><span class="n">size</span><span class="p">()</span> <span class="o">!=</span> <span class="bp">self</span><span class="o">.</span><span class="n">size</span><span class="p">():</span>
<span class="bp">self</span><span class="o">.</span><span class="n">setFixedSize</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">image</span><span class="o">.</span><span class="n">size</span><span class="p">())</span>
<span class="bp">self</span><span class="o">.</span><span class="n">update</span><span class="p">()</span>
<span class="c1"># -----------------------------------------------------------------</span>
</pre></div>
<p>Notice that we call the <tt class="docutils literal">emit</tt> method on the <tt class="docutils literal">face_detected</tt> signal.</p>
<p>But how do we emit arguments? Well we’ll need to define the arguments that we want to pass in our signal. So let’s say that we not only want to emit the fact that we detected a face, but we want to emit the coordinates of the face as well.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">FaceSignalWidget</span><span class="p">(</span><span class="n">FaceDetectionWidget</span><span class="p">):</span>
<span class="n">face_detected</span> <span class="o">=</span> <span class="n">QtCore</span><span class="o">.</span><span class="n">pyqtSignal</span><span class="p">()</span>
<span class="c1"># define our `face_detection_coords` signal</span>
<span class="n">face_detection_coords</span> <span class="o">=</span> <span class="n">QtCore</span><span class="o">.</span><span class="n">pyqtSignal</span><span class="p">(</span><span class="nb">int</span><span class="p">,</span> <span class="nb">int</span><span class="p">,</span> <span class="nb">int</span><span class="p">,</span> <span class="nb">int</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">image_data_slot</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">image_data</span><span class="p">):</span>
<span class="n">faces</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">detect_faces</span><span class="p">(</span><span class="n">image_data</span><span class="p">)</span>
<span class="c1"># If faces our found, `emit` our signal</span>
<span class="k">if</span> <span class="n">faces</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">face_detected</span><span class="o">.</span><span class="n">emit</span><span class="p">()</span>
<span class="k">for</span> <span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">h</span><span class="p">)</span> <span class="ow">in</span> <span class="n">faces</span><span class="p">:</span>
<span class="n">cv2</span><span class="o">.</span><span class="n">rectangle</span><span class="p">(</span><span class="n">image_data</span><span class="p">,</span> <span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">),</span> <span class="p">(</span><span class="n">x</span><span class="o">+</span><span class="n">w</span><span class="p">,</span> <span class="n">y</span><span class="o">+</span><span class="n">h</span><span class="p">),</span> <span class="bp">self</span><span class="o">.</span><span class="n">_red</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_width</span><span class="p">)</span>
<span class="c1"># emit the coordinates, or at least the (x,y), width and height</span>
<span class="bp">self</span><span class="o">.</span><span class="n">face_detection_coords</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">h</span><span class="p">)</span>
<span class="c1"># NOTE: this code is same as base class ----------------------------</span>
<span class="bp">self</span><span class="o">.</span><span class="n">image</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_qimage</span><span class="p">(</span><span class="n">image_data</span><span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">image</span><span class="o">.</span><span class="n">size</span><span class="p">()</span> <span class="o">!=</span> <span class="bp">self</span><span class="o">.</span><span class="n">size</span><span class="p">():</span>
<span class="bp">self</span><span class="o">.</span><span class="n">setFixedSize</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">image</span><span class="o">.</span><span class="n">size</span><span class="p">())</span>
<span class="bp">self</span><span class="o">.</span><span class="n">update</span><span class="p">()</span>
<span class="c1"># -----------------------------------------------------------------</span>
</pre></div>
<p>Note that signals are always defined as class variables instead of instance variables. If you’re confused about the difference, this <a class="reference external" href="https://stackoverflow.com/questions/8959097/what-is-the-difference-between-class-and-instance-variables">stack overflow post</a> does a good job of differentiating the two.</p>
<p>That should be enough to get you started. Be sure to check out the <a class="reference external" href="http://pyqt.sourceforge.net/Docs/PyQt5/signals_slots.html">PyQt documentation on signals and slots</a> for a more in depth treatment.</p>
PyQt Tutorial Index2017-07-20T23:09:00-05:002017-07-20T23:09:00-05:00Ben Hofftag:benhoff.net,2017-07-20:/pyqt-tutorial-index.html<p>Alright, this is top level index showing how to follow the PyQt Tutorial. Feel free to jump in wherever you need to.</p>
<table border="1" class="docutils">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead valign="bottom">
<tr><th class="head">Article</th>
<th class="head">Description</th>
</tr>
</thead>
<tbody valign="top">
<tr><td><a class="reference external" href="https://benhoff.net/pyqt-hello-world.rst">Hello World</a></td>
<td>Create a “Hello World” Graphical User Interface.</td>
</tr>
<tr><td><a class="reference external" href="https://benhoff.net/qt-interface-design.rst">Interface Design</a></td>
<td>Review the three major types of Interfaces.</td>
</tr>
<tr><td><a class="reference external" href="https://benhoff.net/pyqt-layout-design.rst">Layout Mangement</a></td>
<td>Hands on approach how to create …</td></tr></tbody></table><p>Alright, this is top level index showing how to follow the PyQt Tutorial. Feel free to jump in wherever you need to.</p>
<table border="1" class="docutils">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead valign="bottom">
<tr><th class="head">Article</th>
<th class="head">Description</th>
</tr>
</thead>
<tbody valign="top">
<tr><td><a class="reference external" href="https://benhoff.net/pyqt-hello-world.rst">Hello World</a></td>
<td>Create a “Hello World” Graphical User Interface.</td>
</tr>
<tr><td><a class="reference external" href="https://benhoff.net/qt-interface-design.rst">Interface Design</a></td>
<td>Review the three major types of Interfaces.</td>
</tr>
<tr><td><a class="reference external" href="https://benhoff.net/pyqt-layout-design.rst">Layout Mangement</a></td>
<td>Hands on approach how to create Interfaces using
layouts in Qt.</td>
</tr>
<tr><td><a class="reference external" href="https://benhoff.net/pyqt-signals-slots.rst">Signals and Slots</a></td>
<td>Overview of signals and slots</td>
</tr>
</tbody>
</table>
PyQt Layout Design2017-07-17T16:14:00-05:002017-07-17T16:14:00-05:00Ben Hofftag:benhoff.net,2017-07-17:/pyqt-layout-design.html<p class="first last">Let’s work on creating a single interface design face recognition application.</p>
<p>As part of our <a class="reference external" href="https://benhoff.net/pyqt-tutorial.rst">PyQt Tutorial series</a> we’ve built a <a class="reference external" href="https://benhoff.net/pyqt-hello-world.rst">Hello World</a> <span class="caps">GUI</span> application and gone through some basic <a class="reference external" href="https://benhoff.net/qt-interface-design.rst">Interface Design</a> options. In this section, we’re going to implement the first type of Interface, a Single Document Interface. We’re going to create a Face Recognition Application, but since I’d like to focus on the PyQt <span class="caps">GUI</span> development process, the actual face recognition part will be separate from this set of tutorials.</p>
<p>If you’re just jumping in here, check out <a class="reference external" href="https://github.com/benhoff/blog/blob/master/scripts/pyqt-hello-world.py">this link</a> to see the code that we’re starting with.</p>
<p>So let’s install a face recognition ready widget.</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> pip install facerecog
</pre></div>
<p>And then we’ll import it into our program.</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">facerecog</span> <span class="kn">import</span> <span class="n">MainWidget</span><span class="p">,</span> <span class="n">get_haarcascade_filepath</span>
</pre></div>
<p>If you’re interested in seeing the coding going on behind the face recognition piece, feel free to explore it in <a class="reference external" href="https://benhoff.net/face-detection-in-pyqt.rst">this blog post</a>. Otherwise, let’s instantiate our face recognition widget and put it in our program.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">from</span> <span class="nn">PyQt5</span> <span class="kn">import</span> <span class="n">QtCore</span><span class="p">,</span> <span class="n">QtWidgets</span>
<span class="kn">from</span> <span class="nn">facerecog</span> <span class="kn">import</span> <span class="n">MainWidget</span><span class="p">,</span> <span class="n">get_haarcascade_filepath</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="c1"># We need to make the QApplication before our QMainWindow</span>
<span class="c1"># We also need to pass in our system argument values (sys.argv)</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">QtWidgets</span><span class="o">.</span><span class="n">QApplication</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span>
<span class="n">main_window</span> <span class="o">=</span> <span class="n">QtWidgets</span><span class="o">.</span><span class="n">QMainWindow</span><span class="p">()</span>
<span class="c1"># Here's that instantiation bit again.</span>
<span class="n">haar_filepath</span> <span class="o">=</span> <span class="n">get_haarcascade_filepath</span><span class="p">()</span>
<span class="n">main_widget</span> <span class="o">=</span> <span class="n">MainWidget</span><span class="p">(</span><span class="n">haar_filepath</span><span class="p">)</span>
<span class="c1"># QMainWindow requires a central widget. We'll pass in our MainWidget here</span>
<span class="n">main_window</span><span class="o">.</span><span class="n">setCentralWidget</span><span class="p">(</span><span class="n">main_widget</span><span class="p">)</span>
<span class="c1"># Show our main window</span>
<span class="n">main_window</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="c1"># Start the event loop processing</span>
<span class="n">app</span><span class="o">.</span><span class="k">exec</span><span class="p">()</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="n">main</span><span class="p">()</span>
</pre></div>
<img alt="" class="align-center" src="https://benhoff.net/images/face-recognition-success.png" />
<p>Obviously this code isn’t the whole story to the <span class="caps">GUI</span> development. But it demonstrates an important fact about Qt desktop <span class="caps">GUI</span> framework. Everything that the end user interfaces with is a widget. So in this case, since we already have a ready made widget, we can plug it in and send our user on their merry way.</p>
<p>The end goal for most of our interface design then using the Qt framework will be to wrap our interface into widgets. So how do we do that?</p>
<p>Well, as we talked about in the Hello World blog post, we can add in some functionality of a menu bar, status bar, dock widgets, etc.</p>
<img alt="" class="align-center" src="http://doc.qt.io/qt-5/images/mainwindowlayout.png" />
<p>But most of the time what we want to do is group, or layout, multiple widgets in an area. The way to do that in Qt is through Layout Management.</p>
<p>All QWidget classes use layouts to manage the life cycle of their subwidgets. In addition to managing the life cycle, ayouts also:</p>
<ul class="simple">
<li>Positioning of child widgets.</li>
<li>Sensible default sizes for windows.</li>
<li>Sensible minimum sizes for windows.</li>
<li>Resize handling.</li>
<li>Automatic updates when contents change:<ul>
<li>Font size, text or other contents of child widgets.</li>
<li>Hiding or showing a child widget.</li>
<li>Removal of child widgets.</li>
</ul>
</li>
</ul>
<p>Now to be clear, the Qt Documentation does an better job then I will of explaining the in’s and out’s of layout management, so I’d recommend you <a class="reference external" href="http://doc.qt.io/qt-5/layout.html">check it out</a>. But we’ll go through the basics here of instantiating, using and setting layouts here.</p>
<p>First we need a master widget that we can set the layout on.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">QtWidgets</span>
<span class="k">class</span> <span class="nc">MasterWidget</span><span class="p">(</span><span class="n">QWidget</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="n">parent</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">parent</span><span class="p">)</span>
</pre></div>
<p>Then we need to pick which layout that we want to use.</p>
<p>Some of my favorite layouts include <tt class="docutils literal">QHBoxLayout</tt> which arranges widgets horizontally, <tt class="docutils literal">QVBoxLayout</tt>, which arranges widgets vertically, and <tt class="docutils literal">QGridLayout</tt> which allows you to put things in a grid.</p>
<p>Let’s use a layout to stack a couple of widgets on top of each other. We’ll do this using the <tt class="docutils literal">addWidget</tt> function on our instantiated layout. Once we’ve added all of the widgets that we want, we’ll then set the layout using the <tt class="docutils literal">setLayout</tt> method on our <tt class="docutils literal">MasterWidget</tt> class.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">QtWidgets</span>
<span class="k">class</span> <span class="nc">MasterWidget</span><span class="p">(</span><span class="n">QtWidgets</span><span class="o">.</span><span class="n">QWidget</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">parent</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">parent</span><span class="p">)</span>
<span class="c1"># Create a label, push button and line edit widgets</span>
<span class="n">label</span> <span class="o">=</span> <span class="n">QtWidgets</span><span class="o">.</span><span class="n">QLabel</span><span class="p">(</span><span class="s1">'This is a label'</span><span class="p">)</span>
<span class="n">run_push_button</span> <span class="o">=</span> <span class="n">QtWidgets</span><span class="o">.</span><span class="n">QPushButton</span><span class="p">(</span><span class="s1">'Click Me'</span><span class="p">)</span>
<span class="n">line_edit</span> <span class="o">=</span> <span class="n">QtWidgets</span><span class="o">.</span><span class="n">QLineEdit</span><span class="p">(</span><span class="s1">'Default line edit text'</span><span class="p">)</span>
<span class="c1"># create our layout, a vertical layout</span>
<span class="n">layout</span> <span class="o">=</span> <span class="n">QtWidgets</span><span class="o">.</span><span class="n">QVBoxLayout</span><span class="p">()</span>
<span class="c1"># add widgets</span>
<span class="n">layout</span><span class="o">.</span><span class="n">addWidget</span><span class="p">(</span><span class="n">label</span><span class="p">)</span>
<span class="n">layout</span><span class="o">.</span><span class="n">addWidget</span><span class="p">(</span><span class="n">line_edit</span><span class="p">)</span>
<span class="n">layout</span><span class="o">.</span><span class="n">addWidget</span><span class="p">(</span><span class="n">run_push_button</span><span class="p">)</span>
<span class="c1"># set the layout of our master widget</span>
<span class="bp">self</span><span class="o">.</span><span class="n">setLayout</span><span class="p">(</span><span class="n">layout</span><span class="p">)</span>
</pre></div>
<p>This is the resulting gui for the above code, showing all of the widgets stacked vertically.</p>
<img alt="" class="align-center" src="https://benhoff.net/images/pyqt-layout-example.png" />
<p>We could also use a grid layout. The <span class="caps">API</span> for that is a little different. Whenever we add a widget, we need to pass in the row and the column. I’m going to do this using key word arguments so that you can see what is going on. If we remove the <tt class="docutils literal">row</tt> and <tt class="docutils literal">column</tt> key word arguments, this would still be valid code, as these are the positional arguments as well.</p>
<div class="highlight"><pre><span></span><span class="n">layout</span> <span class="o">=</span> <span class="n">QtWidgets</span><span class="o">.</span><span class="n">QGridLayout</span><span class="p">()</span>
<span class="c1"># add as row 0, column 0</span>
<span class="n">layout</span><span class="o">.</span><span class="n">addWidget</span><span class="p">(</span><span class="n">label</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="c1"># add as row 1, column 0</span>
<span class="n">layout</span><span class="o">.</span><span class="n">addWidget</span><span class="p">(</span><span class="n">line_edit</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="c1"># add as row 1, column 1</span>
<span class="n">layout</span><span class="o">.</span><span class="n">addWidget</span><span class="p">(</span><span class="n">run_push_button</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
</pre></div>
<p>Below, you can see the change in the appearance of our <span class="caps">GUI</span> with the above layout changes.</p>
<img alt="" class="align-center" src="https://benhoff.net/images/pyqt-layout-example-2.png" />
<p>Layouts are a powerful way to add more widgets and control how those new widgets look in our <span class="caps">GUI</span>, so don’t be afraid to use them!</p>
<div class="section" id="wrapup">
<h2>Wrapup</h2>
<p>You can catch the source code for the tutorial up to this point <a class="reference external" href="https://github.com/benhoff/blog/blob/master/scripts/pyqt-layout-design.py">here</a>.</p>
<p>There’s only one problem with our <span class="caps">GUI</span> currently. If we click that push button, nothing happens. What’s up with that? Well we need to connect our signals and slots together. Check out how to do that, in the <a class="reference external" href="https://benhoff.net/pyqt-signals-slots.rst">Signals and Slots</a> post. Or, if you’re already familiar with how signals and slots work, feel free to skip ahead using <a class="reference external" href="https://benhoff.net/pyqt-tutorial.rst">this index</a> of the entire tutorial.</p>
</div>
Face Detection with OpenCV and PyQt2017-07-14T19:37:00-05:002017-07-14T19:37:00-05:00Ben Hofftag:benhoff.net,2017-07-14:/face-detection-opencv-pyqt.html<p class="first last">Getting Face Detection working with OpenCV and PyQt</p>
<p>Let’s install some stuff.</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> pip install opencv-python numpy PyQt5
</pre></div>
<p>Let’s import some stuff.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">from</span> <span class="nn">os</span> <span class="kn">import</span> <span class="n">path</span>
<span class="kn">import</span> <span class="nn">cv2</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">PyQt5</span> <span class="kn">import</span> <span class="n">QtCore</span>
<span class="kn">from</span> <span class="nn">PyQt5</span> <span class="kn">import</span> <span class="n">QtWidgets</span>
<span class="kn">from</span> <span class="nn">PyQt5</span> <span class="kn">import</span> <span class="n">QtGui</span>
</pre></div>
<p>Now we’ll build this backwards, starting with the smallest pieces and working bigger. Let’s start working on creating the video recording and getting the face coordinates out.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">RecordVideo</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">camera_port</span><span class="o">=</span><span class="mi">0</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">camera</span> <span class="o">=</span> <span class="n">cv2</span><span class="o">.</span><span class="n">VideoCapture</span><span class="p">(</span><span class="n">camera_port</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">running</span> <span class="o">=</span> <span class="bp">False</span>
<span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">running</span> <span class="o">=</span> <span class="bp">True</span>
<span class="k">while</span> <span class="bp">self</span><span class="o">.</span><span class="n">running</span><span class="p">:</span>
<span class="n">read</span><span class="p">,</span> <span class="n">image</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">camera</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="c1"># TODO: detect faces now</span>
</pre></div>
<p>Awesome, now let’s work on the facial detection.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">FaceDetection</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">haar_cascade_filepath</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">classifier</span> <span class="o">=</span> <span class="n">cv2</span><span class="o">.</span><span class="n">CascadeClassifier</span><span class="p">(</span><span class="n">haar_cascade_filepath</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_min_size</span> <span class="o">=</span> <span class="p">(</span><span class="mi">30</span><span class="p">,</span> <span class="mi">30</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">detect_faces</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">image</span><span class="p">):</span>
<span class="c1"># haarclassifiers work better in black and white</span>
<span class="n">gray_image</span> <span class="o">=</span> <span class="n">cv2</span><span class="o">.</span><span class="n">cvtColor</span><span class="p">(</span><span class="n">image</span><span class="p">,</span> <span class="n">cv2</span><span class="o">.</span><span class="n">COLOR_BGR2GRAY</span><span class="p">)</span>
<span class="n">gray_image</span> <span class="o">=</span> <span class="n">cv2</span><span class="o">.</span><span class="n">equalizeHist</span><span class="p">(</span><span class="n">grey_image</span><span class="p">)</span>
<span class="n">faces</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">classifier</span><span class="o">.</span><span class="n">detectMultiScale</span><span class="p">(</span><span class="n">grey_image</span><span class="p">,</span> <span class="n">scaleFactor</span><span class="o">=</span><span class="mf">1.3</span><span class="p">,</span> <span class="n">minNeighbors</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">flags</span><span class="o">=</span><span class="n">cv2</span><span class="o">.</span><span class="n">CASCADE_SCALE_IMAGE</span><span class="p">,</span> <span class="n">min_size</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_min_size</span><span class="p">)</span>
<span class="c1"># TODO: Paint on a surface and add the faces.</span>
</pre></div>
<p>Ok so we need something to paint on. This is where we need to switch gears a little bit. We’re going to use Qt to paint on. So we’ll need to rework some of classes to play nicely with Qt.</p>
<p>We’ll start by making our <tt class="docutils literal">RecordVideo</tt> a subclass of <tt class="docutils literal">QObject</tt>. We’ll also create a signal called <tt class="docutils literal">image_data</tt> and have it emit a <tt class="docutils literal">np.ndarray</tt> in the <tt class="docutils literal">timerEvent</tt>. We’ll use it in the <tt class="docutils literal">timerEvent</tt> so that we can keep it single threaded.</p>
<p>If that sounds confusing, don’t worry. The code isn’t that long.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">RecordVideo</span><span class="p">(</span><span class="n">QtCore</span><span class="o">.</span><span class="n">QObject</span><span class="p">):</span>
<span class="n">image_data</span> <span class="o">=</span> <span class="n">QtCore</span><span class="o">.</span><span class="n">pyqtSignal</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">ndarray</span><span class="p">)</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">camera_port</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">parent</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">parent</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">camera</span> <span class="o">=</span> <span class="n">cv2</span><span class="o">.</span><span class="n">VideoCapture</span><span class="p">(</span><span class="n">camera_port</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">timer</span> <span class="o">=</span> <span class="n">QtCore</span><span class="o">.</span><span class="n">QBasicTimer</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">start_recording</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">timer</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">timerEvent</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">event</span><span class="p">):</span>
<span class="k">if</span> <span class="p">(</span><span class="n">event</span><span class="o">.</span><span class="n">timerId</span><span class="p">()</span> <span class="o">!=</span> <span class="bp">self</span><span class="o">.</span><span class="n">timer</span><span class="o">.</span><span class="n">timerId</span><span class="p">()):</span>
<span class="k">return</span>
<span class="n">read</span><span class="p">,</span> <span class="n">image</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">camera</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="k">if</span> <span class="n">read</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">image_ready</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="n">image</span><span class="p">)</span>
</pre></div>
<p>We need to extend our Face Detection as well. We’ll make it a <tt class="docutils literal">QWidget</tt>, because we want to paint on it. We’ll add a new method that converts our ndarray into a <tt class="docutils literal">QImage</tt>. The painting loop occasionally might be too slow, so we’ll save the image to the class and call the <tt class="docutils literal">update</tt> method at the end of the slot. We’ll draw the rectangle on into the data using the <tt class="docutils literal">cv2.rectangle</tt> method call.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">FaceDetectionWidget</span><span class="p">(</span><span class="n">QtWidgets</span><span class="o">.</span><span class="n">QWidget</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">haar_cascade_filepath</span><span class="p">,</span> <span class="n">parent</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">parent</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">classifier</span> <span class="o">=</span> <span class="n">cv2</span><span class="o">.</span><span class="n">CascadeClassifier</span><span class="p">(</span><span class="n">haar_cascade_filepath</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">image</span> <span class="o">=</span> <span class="n">QtGui</span><span class="o">.</span><span class="n">QImage</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_red</span> <span class="o">=</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">255</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_width</span> <span class="o">=</span> <span class="mi">2</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_min_size</span> <span class="o">=</span> <span class="p">(</span><span class="mi">30</span><span class="p">,</span> <span class="mi">30</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">detect_faces</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">image</span><span class="p">:</span> <span class="n">np</span><span class="o">.</span><span class="n">ndarray</span><span class="p">):</span>
<span class="c1"># haarclassifiers work better in black and white</span>
<span class="n">gray_image</span> <span class="o">=</span> <span class="n">cv2</span><span class="o">.</span><span class="n">cvtColor</span><span class="p">(</span><span class="n">image</span><span class="p">,</span> <span class="n">cv2</span><span class="o">.</span><span class="n">COLOR_BGR2GRAY</span><span class="p">)</span>
<span class="n">gray_image</span> <span class="o">=</span> <span class="n">cv2</span><span class="o">.</span><span class="n">equalizeHist</span><span class="p">(</span><span class="n">gray_image</span><span class="p">)</span>
<span class="n">faces</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">classifier</span><span class="o">.</span><span class="n">detectMultiScale</span><span class="p">(</span><span class="n">gray_image</span><span class="p">,</span> <span class="n">scaleFactor</span><span class="o">=</span><span class="mf">1.3</span><span class="p">,</span> <span class="n">minNeighbors</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">flags</span><span class="o">=</span><span class="n">cv2</span><span class="o">.</span><span class="n">CASCADE_SCALE_IMAGE</span><span class="p">,</span> <span class="n">minSize</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_min_size</span><span class="p">)</span>
<span class="k">return</span> <span class="n">faces</span>
<span class="k">def</span> <span class="nf">image_data_slot</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">image_data</span><span class="p">):</span>
<span class="n">faces</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">detect_faces</span><span class="p">(</span><span class="n">image_data</span><span class="p">)</span>
<span class="k">for</span> <span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">h</span><span class="p">)</span> <span class="ow">in</span> <span class="n">faces</span><span class="p">:</span>
<span class="n">cv2</span><span class="o">.</span><span class="n">rectangle</span><span class="p">(</span><span class="n">image_data</span><span class="p">,</span> <span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">),</span> <span class="p">(</span><span class="n">x</span><span class="o">+</span><span class="n">w</span><span class="p">,</span> <span class="n">y</span><span class="o">+</span><span class="n">h</span><span class="p">),</span> <span class="bp">self</span><span class="o">.</span><span class="n">_red</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_width</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">image</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_qimage</span><span class="p">(</span><span class="n">image_data</span><span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">image</span><span class="o">.</span><span class="n">size</span><span class="p">()</span> <span class="o">!=</span> <span class="bp">self</span><span class="o">.</span><span class="n">size</span><span class="p">():</span>
<span class="bp">self</span><span class="o">.</span><span class="n">setFixedSize</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">image</span><span class="o">.</span><span class="n">size</span><span class="p">())</span>
<span class="bp">self</span><span class="o">.</span><span class="n">update</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">get_qimage</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">image</span><span class="p">:</span> <span class="n">np</span><span class="o">.</span><span class="n">ndarray</span><span class="p">):</span>
<span class="n">height</span><span class="p">,</span> <span class="n">width</span><span class="p">,</span> <span class="n">colors</span> <span class="o">=</span> <span class="n">image</span><span class="o">.</span><span class="n">shape</span>
<span class="n">bytesPerLine</span> <span class="o">=</span> <span class="mi">3</span> <span class="o">*</span> <span class="n">width</span>
<span class="n">QImage</span> <span class="o">=</span> <span class="n">QtGui</span><span class="o">.</span><span class="n">QImage</span>
<span class="n">image</span> <span class="o">=</span> <span class="n">QImage</span><span class="p">(</span><span class="n">image</span><span class="o">.</span><span class="n">data</span><span class="p">,</span> <span class="n">width</span><span class="p">,</span> <span class="n">height</span><span class="p">,</span> <span class="n">bytesPerLine</span><span class="p">,</span> <span class="n">QImage</span><span class="o">.</span><span class="n">Format_RGB888</span><span class="p">)</span>
<span class="n">image</span> <span class="o">=</span> <span class="n">image</span><span class="o">.</span><span class="n">rgbSwapped</span><span class="p">()</span>
<span class="k">return</span> <span class="n">image</span>
<span class="k">def</span> <span class="nf">paintEvent</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">event</span><span class="p">):</span>
<span class="n">painter</span> <span class="o">=</span> <span class="n">QtGui</span><span class="o">.</span><span class="n">QPainter</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
<span class="n">painter</span><span class="o">.</span><span class="n">drawImage</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">image</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">image</span> <span class="o">=</span> <span class="n">QtGui</span><span class="o">.</span><span class="n">QImage</span><span class="p">()</span>
</pre></div>
<p>Now let’s tie these two together. We’ll create a MainWidget class that instantiates both of our two custom classes, puts our widget in a Layout along with a start button. We also need to tie together both the image signal/slot and the click’d and start recording method.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">MainWidget</span><span class="p">(</span><span class="n">QtWidgets</span><span class="o">.</span><span class="n">QWidget</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">haarcascade_filepath</span><span class="p">,</span> <span class="n">parent</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">parent</span><span class="p">)</span>
<span class="n">fp</span> <span class="o">=</span> <span class="n">haarcascade_filepath</span>
<span class="bp">self</span><span class="o">.</span><span class="n">face_detection_widget</span> <span class="o">=</span> <span class="n">FaceDetectionWidget</span><span class="p">(</span><span class="n">fp</span><span class="p">)</span>
<span class="c1"># TODO: set video port</span>
<span class="bp">self</span><span class="o">.</span><span class="n">record_video</span> <span class="o">=</span> <span class="n">RecordVideo</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">run_button</span> <span class="o">=</span> <span class="n">QtWidgets</span><span class="o">.</span><span class="n">QPushButton</span><span class="p">(</span><span class="s1">'Start'</span><span class="p">)</span>
<span class="c1"># Connect the image data signal and slot together</span>
<span class="n">image_data_slot</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">face_detection_widget</span><span class="o">.</span><span class="n">image_data_slot</span>
<span class="bp">self</span><span class="o">.</span><span class="n">record_video</span><span class="o">.</span><span class="n">image_data</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">image_data_slot</span><span class="p">)</span>
<span class="c1"># connect the run button to the start recording slot</span>
<span class="bp">self</span><span class="o">.</span><span class="n">run_button</span><span class="o">.</span><span class="n">clicked</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">record_video</span><span class="o">.</span><span class="n">start_recording</span><span class="p">)</span>
<span class="c1"># Create and set the layout</span>
<span class="n">layout</span> <span class="o">=</span> <span class="n">QtWidgets</span><span class="o">.</span><span class="n">QVBoxLayout</span><span class="p">()</span>
<span class="n">layout</span><span class="o">.</span><span class="n">addWidget</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">face_detection_widget</span><span class="p">)</span>
<span class="n">layout</span><span class="o">.</span><span class="n">addWidget</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">run_button</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">setLayout</span><span class="p">(</span><span class="n">layout</span><span class="p">)</span>
</pre></div>
<p>Now we just need to create our QApplication, QMainWindow, and our custom MainWidget.</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">haar_cascade_filepath</span><span class="p">):</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">QtWidgets</span><span class="o">.</span><span class="n">QApplication</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span>
<span class="n">main_window</span> <span class="o">=</span> <span class="n">QtWidgets</span><span class="o">.</span><span class="n">QMainWindow</span><span class="p">()</span>
<span class="n">main_widget</span> <span class="o">=</span> <span class="n">MainWidget</span><span class="p">(</span><span class="n">haar_cascade_filepath</span><span class="p">)</span>
<span class="n">main_window</span><span class="o">.</span><span class="n">setCentralWidget</span><span class="p">(</span><span class="n">main_widget</span><span class="p">)</span>
<span class="n">main_window</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="n">app</span><span class="o">.</span><span class="n">exec_</span><span class="p">())</span>
</pre></div>
<p>Now we need to pass in the haarcascade file. I saved it using this shell script.</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> curl https://raw.githubusercontent.com/opencv/opencv/master/data/haarcascades/haarcascade_frontalface_default.xml > haarcascade_frontalface_default.xml
</pre></div>
<p>You’ll have to change the code to match your local setup.</p>
<p>I’ll distribute this out on pypi so you can install it eventually.</p>
<div class="highlight"><pre><span></span><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="n">script_dir</span> <span class="o">=</span> <span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">path</span><span class="o">.</span><span class="n">realpath</span><span class="p">(</span><span class="vm">__file__</span><span class="p">))</span>
<span class="n">cascade_filepath</span> <span class="o">=</span> <span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">script_dir</span><span class="p">,</span> <span class="s1">'..'</span><span class="p">,</span> <span class="s1">'data'</span><span class="p">,</span> <span class="s1">'haarcascade_frontalface_default.xml'</span><span class="p">)</span>
<span class="n">cascade_filepath</span> <span class="o">=</span> <span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="n">cascade_filepath</span><span class="p">)</span>
<span class="n">main</span><span class="p">(</span><span class="n">cascade_filepath</span><span class="p">)</span>
</pre></div>
<img alt="" src="https://benhoff.net/images/face-recognition-success.png" />
<p>Success!</p>
<p>You can see all the source code <a class="reference external" href="https://github.com/benhoff/blog/blob/master/scripts/face-detection-in-pyqt.py">here</a>.</p>
Setup Mailgun with Google Domain2017-07-05T16:24:00-05:002017-07-05T16:24:00-05:00Ben Hofftag:benhoff.net,2017-07-05:/setup-mailgun-with-google-domain.html<p class="first last">Setting up the <span class="caps">DNS</span> configuration of Mailgun for Google Domain</p>
<p>Right, let’s setup Mailgun. I’m using Mailgun for my Discourse install.</p>
<p>The first step is to add your domain.</p>
<p>Mailgun is going to recommend that you use a subdomain, such as <em>mg.mydomain.com</em>
Whether you want to do that or not is up to you, the steps will be similar. I’m not going to. So for the sake of argument here, the domain that we are setting up is <em>mydomain.com</em></p>
<p>Mailgun is going to give you 4 steps. We will need to go <a class="reference external" href="https://domains.google/#/">Google Domains</a>. Click the <em>Manage My Domains</em>, and then click the <em><span class="caps">DNS</span></em> button on the domain of interest.</p>
<p>Once you’re there, scroll all the way down to the section that is titled <em>Custom resource records</em>. It’ll look something like this:</p>
<img alt="" src="https://benhoff.net/images/google-domain-1.png" style="width: 400px; height: 100px;" />
<p>That accomplishes Step 1. <em>Go To Your <span class="caps">DNS</span> Provider</em> from the 4 step process in Mailgun.</p>
<p>Under Step 2, <em>Add <span class="caps">DNS</span> Records For Sending</em> on the <strong>Mailgun</strong> site, there are three resources we need to add: <tt class="docutils literal"><span class="caps">TXT</span></tt>, <tt class="docutils literal"><span class="caps">MX</span></tt>, and a second <tt class="docutils literal"><span class="caps">MX</span></tt>.</p>
<p>Let’s add the <tt class="docutils literal"><span class="caps">TXT</span></tt> first.</p>
<p>The top bar that is slightly grayed out with a blue button “Add” button on the far right is the interface that we’ll be using to add these documents. If you have a subdomain, (e.g. “mg.mydomain.com”), the subdomain will go in the far left “Name” column, otherwise leave it blank. The “Type” drop down menu (second column from the left), we’ll change from “A” to “<span class="caps">TXT</span>”. You can leave the “<span class="caps">TTL</span>” column as is. In the “Data” column, we’ll put in the <span class="caps">SPF</span> value “v=spf1 include:mailgun.org ~all”. Hit Add, and you should see the table update below the top bar.</p>
<p>Next, we’ll add in the <span class="caps">DKIM</span>, or Domain Key. We’ll follow the same procedure, although this time you’ll need to add in the Hostname that Mailgun specifies. Change the “Type” from “A” to “<span class="caps">TXT</span>” and paste the value associated with the domain key into the “Data” column (should be in the form of <tt class="docutils literal"><span class="pre">k=rsa;p={a</span> bunch of letters}</tt>).</p>
<p>Next we’ll add in the <tt class="docutils literal"><span class="caps">MX</span></tt> records.</p>
<p>We’ll follow the same drill, subdomain (if you have one) in the far left, change the “Type” from “A” to “<span class="caps">MX</span>” and then paste the <tt class="docutils literal">mxa.mailgun.org.</tt> in the. You can either hit the plus sign on the right hand side of the “Data” column, or hit “Add” and then scroll down to the entry and click the “Edit” button. As a forewarning, Google will automatically add the number 10 to your record so <tt class="docutils literal">mxa.mailgun.org.</tt> will change to <tt class="docutils literal">10 mxa.mailgun.org.</tt>. Totally legit, don’t panic.</p>
<p>Add in the second “<span class="caps">MX</span>” record <tt class="docutils literal">mxb.mailgun.org.</tt> and “Add” or save the record.</p>
<p>Last, according to Step 3) we need to Add <span class="caps">DNS</span> Records for Tracking.</p>
<p>Copy the Hostname that Mailgun provides under Step 3, change the “Type” from “A” to “<span class="caps">CNAME</span>” and enter in the value of <tt class="docutils literal">mailgun.org</tt> into the “Data” column. Click add.</p>
<p>Step 4 is wait for your domain to verify. In my experience, it was almost instantaneous with Google.</p>
<p>In the end it should look something (although not exactly) like this.</p>
<img alt="" src="https://benhoff.net/images/google-domain-2.png" style="width: 600px; height: 424px;" />
Vexbot State of the Union2017-07-01T16:30:00-05:002017-07-01T16:30:00-05:00Ben Hofftag:benhoff.net,2017-07-01:/vexbot-state-of-the-union.html<p>All right, let’s talk about <a class="reference external" href="https://github.com/benhoff/vexbot">Vexbot</a>.</p>
<p>Vexbot was created to scratch a very specific itch, conglomerating multiple chat sources into one place. No other bot (to my knowledge) can do this <a class="footnote-reference" href="#id3" id="id1">[1]</a>.</p>
<p>Vexbot does this by launching a subprocess for each chat provider (<span class="caps">IRC</span>, <span class="caps">XMPP</span>, Youtube, etc.) and using …</p><p>All right, let’s talk about <a class="reference external" href="https://github.com/benhoff/vexbot">Vexbot</a>.</p>
<p>Vexbot was created to scratch a very specific itch, conglomerating multiple chat sources into one place. No other bot (to my knowledge) can do this <a class="footnote-reference" href="#id3" id="id1">[1]</a>.</p>
<p>Vexbot does this by launching a subprocess for each chat provider (<span class="caps">IRC</span>, <span class="caps">XMPP</span>, Youtube, etc.) and using messaging (via <a class="reference external" href="http://zeromq.org/">zmq</a>) to communicate between them all. The <span class="caps">GUI</span> application, <a class="reference external" href="https://github.com/benhoff/CHATIMUSMAXIMUS">chatimusmaximus</a> provides the user interface so that I could see all of the chat streams in one central place.</p>
<p>Vexbot has some simple chat processing, including direct string matching, and some natural language processing using the bag of words method to guess intent. These pieces are significantly underdeveloped. The reason that these important pieces are underdeveloped is because a large amount of recent developer time has has been spent devoted to two important pieces: settings and process management <a class="footnote-reference" href="#id4" id="id2">[2]</a>.</p>
<p>For settings management, I was originally using <a class="reference external" href="http://www.yaml.org/start.html"><span class="caps">YAML</span></a> to manage all the settings. I really liked this approach as it allowed me to leave comments in the configuration file for my users. For a simple stationary setup, this would work well. However, for changing values programmatically, the limitations of this system were quickly apparent. I wanted to create a quick setup feature where the setup could be programmed from the command line, likely using a Text User Interface (<span class="caps">TUI</span>). <span class="caps">YAML</span> files aren’t the best format for doing this, frankly.</p>
<p>I started exploring <a class="reference external" href="https://www.sqlalchemy.org/">SQLALchemy</a> for this purpose, but have yet to finish up an implementation that I’m happy with. The current idea is that more “dynamic” services (<span class="caps">IRC</span>, <span class="caps">XMPP</span>) will get their own table for settings. Other, more static services will use the <a class="reference external" href="https://en.wikipedia.org/wiki/INI_file"><span class="caps">INI</span></a> format. The python standard library <a class="reference external" href="https://docs.python.org/3/library/configparser.html">configparser</a> library handles the <span class="caps">INI</span> format nicely, it’s easy enough to write out programmatically, and common to rewrite using a text editor.</p>
<p>So now I’ve got an <span class="caps">SQL</span> database and a configuration file that I need to unify into one easy to use <span class="caps">API</span>. I also need to settle on schema that I like. It’ll take work, but I’m happy enough with the work.</p>
<p>The thing I’m not happy about is the process management piece.</p>
<p>Right not the process management is done in python, using the <a class="reference external" href="https://docs.python.org/3/library/subprocess.html">subprocess</a> module. And I’m really not happy about it.</p>
<p>The point of Vexbot is not to duplicate any functionality that I have to. And man, have you ever noticed how good operating systems are at process management? Problem being that operating systems are hilariously un-cross platform.</p>
<p>I would probably double down and use systemd, but it forces me to bump up to the system level, instead of staying in userland. It’s an unsolved issue. For now, I’ll stick with using as much as the subprocess module as possible. Start, stop, kill, restart, update processes. The whole shebang.</p>
<p>Once the settings and the process management piece is taken care of, I’ll get back to the natural language processing and expanding the plugins.</p>
<table class="docutils footnote" frame="void" id="id3" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>The next closest well supported piece of software that I could find in Python was <a class="reference external" href="https://github.com/errbotio/errbot">errbot</a>. Errbot had some limitations though, mainly only allowing a single chat provider at a time. Errbot does allow one thing that Vexbot does not, a unified <span class="caps">API</span> for chat related programming regardless of backend.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id4" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>It should be noted that the inital push was to get simple one-way text communication working between the various supported text services. Two way communication, or in-service commands haven’t been, and continue to not be, a priority.</td></tr>
</tbody>
</table>
Keeping Track of Past Research2017-06-27T17:07:00-05:002017-06-27T17:07:00-05:00Ben Hofftag:benhoff.net,2017-06-27:/keeping-track-of-past-research.html<p class="first last">Reviewing multiple methods of keeping track of past research</p>
<p>I’ve got a very specific problem.</p>
<p>While I was doing research as part of my <a class="reference external" href="https://www.packtpub.com/big-data-and-business-intelligence/learning-python-data-analysis-video">Learning Python Data Analysis</a> series, I stumbled across a slideshare that showed the different types of analysis that you can do using Twitter data. I think the author ended up deep diving into building a network diagram showing the different connections between tweets. I want to deep dive into a different aspect of Twitter data analysis, following on the trend of doing live analysis that I did in my aforementioned video series is.</p>
<p>My problem is that I can’t find that slideshare.</p>
<p>Now I admittedly have done very little research to re-find it. I actually don’t feel like doing any research though. I would like for it to be served to me on a silver platter, since I already spent probably 10+ hours Google searching.</p>
<p>So I went looking for a better way.</p>
<p>There’s a couple of different ways I’ve found.</p>
<ul class="simple">
<li>Mind maps</li>
<li>Note cards</li>
<li>Wiki’s</li>
<li>Blogs</li>
</ul>
<div class="section" id="mind-maps">
<h2>Mind Maps</h2>
<p>Mind maps work by showing the connections between different pieces.</p>
<img alt="" src="https://upload.wikimedia.org/wikipedia/commons/1/1a/Tennis-mindmap.png" style="width: 400px; height: 282px;" />
<p>I’ve been aware of mind maps before, but I was recently reminded of them by an interactive mind map I saw on Github, promising/labeled “Learn Anything”. Here’s the <a class="reference external" href="https://github.com/nikitavoloboev/learn-anything">repo</a> and here’s the <a class="reference external" href="https://learn-anything.xyz/">website</a> if you want to take a look.</p>
<p>I’ll be upfront, I don’t really see the use in mind maps… or at least not software generated ones. The Learn Anything site seems to be more a link map, which I can kind of buy for usefulness? Kinda?</p>
</div>
<div class="section" id="note-cards">
<h2>Note Cards</h2>
<p>This is a method I was thought while I was in school, and is apparently still in use today. The basic gist is to record the idea or information along with the original source on note cards. So you might pull the idea of “Elephants can hear in the infrasonic region” out of a certain book. The useful point of note cards is not to record the idea, but the actual source. That way, when you were writing a research paper you could form a coherent paper/thought process without worrying about keeping track of the sources, as they were a standalone product.</p>
<p>Microsoft Word has a version of this feature that I have [ab]used to write reports before. There are also standalone products that do a version of this as well. The challenge is that webpages, unlike books, do not fit well within this format. <a class="reference external" href="https://en.wikipedia.org/wiki/Link_rot">Link rot</a> is also a concern with this method.</p>
</div>
<div class="section" id="wiki-s">
<h2>Wiki’s</h2>
<p>I’ve been interested in the idea of creating wiki’s for a long time. My interests are diverse, and the idea of hooking a wiki site up to a search engine is very, very appealing. <a class="reference external" href="http://calnewport.com/blog/2009/05/11/how-to-build-a-paper-research-wiki/">This Blog post</a> explores in depth a particular research methodology the author recommends as key to enabling his research.</p>
<p>However, according to Wikipedia, a wiki is a website where users collaboratively edit articles. In this case, keeping track of individual lines of research isn’t really a collaborative effort. Wikipedia also mentions that wiki’s are basically Content Management Systems (<span class="caps">CMS</span>’s), which is a pretty well worn software model. It would likely be possible to explore a open source <span class="caps">CMS</span> for this solution. Or you know, maybe a blog….</p>
</div>
<div class="section" id="blogs">
<h2>Blogs</h2>
<p>Blogs at the end of the day are just content and can say anything. One of my favorite posts so far has been the <a class="reference external" href="https://benhoff.net/flashing-beaglebone.rst">Beaglebone Black</a> post, where I walked through the steps and frustrations of flashing a Beaglebone black. While I was originally trying to install OpenVPN on the Beaglebone and ended up not doing that, the post serves as a reference point when I (invariably) reflash another one. But I’ve got some interesting tidbits and posts that should save me some time… next time.</p>
</div>
<div class="section" id="closing-thoughts">
<h2>Closing Thoughts</h2>
<p>The truth of the matter is there’s likely no single model or silver bullet software that is going to make this easy for me. My use case falls somewhere between <a class="reference external" href="https://en.wikipedia.org/wiki/Knowledge_transfer">Knowledge Transfer</a> and individual research.</p>
<p>Committing to a research methodology or note taking process would be the main piece. Since I’ve enjoyed writing up blog posts and linking them to death with outside sources, I’ll likely keep that approach until further notice.</p>
<p>On the software side, I would like to figure out a way to keep track of interesting links and download/index the page contents to stave off the issues associated with link rot. Hooking this up to a search engine, (likely elastic search) combined with the aforementioned blog posts I and my video series transcript would scratch the 80% solution. If you know of an easy way to do that, drop me a line.</p>
</div>
Qt Interface Design2017-06-26T17:21:00-05:002017-06-26T17:21:00-05:00Ben Hofftag:benhoff.net,2017-06-26:/qt-interface-design.html<p class="first last">Reviewing types of interface design for Qt</p>
<p>As part of our <a class="reference external" href="https://benhoff.net/pyqt-tutorial.rst">PyQt Tutorial series</a>, we’ve got a <a class="reference external" href="https://benhoff.net/pyqt-hello-world.rst">hello world gui app</a> running. Now we need to design our application. So let’s talk about some general desktop <span class="caps">GUI</span> design. You might have used a lot of <span class="caps">GUI</span> applications, but you probably never noticed their design. Especially (and super ironically) if it was done well. As a fun exercise, if someone asked you to design a web browser right now, what would your interface look like?</p>
<p>If you’re stumped, don’t worry. A lot of people have thought about this before and have ideas. Three main ones in fact.</p>
<ul class="simple">
<li>Single Document Interface</li>
<li>Tabbed Document Interface</li>
<li>Multiple Document Interface</li>
</ul>
<div class="section" id="single-document-interfaces">
<h2>Single Document Interfaces</h2>
<p>In a Single Document Interface, there’s only a single window for each instance of the application. LibreOffice Writer, Microsoft Word, and most other word processors share this design.</p>
<img alt="" src="https://benhoff.net/images/libreoffice-snap.png" />
<p>Each document, or application, that’s opened gets it’s own window. The window management is done by the native Desktop Environment.</p>
<p>I’d recommend a single document interface for most GUIs as long as the functionality is simple. Make sure that when you’re designing your gui that pulling up multiple instances won’t lock up resources (ports, databases, files, etc.).</p>
<!-- TODO add in some example code of a single document design. -->
<p>Single document design in PyQt is the default. We’ve pretty much got it going in our hello world example. The only thing we need to do is add some content to our Gui and organize it. I typically subclass a <tt class="docutils literal">QWidget</tt> and use <tt class="docutils literal">QLayout</tt> with <tt class="docutils literal">QWidget.setLayout</tt> function to group things logically. <tt class="docutils literal">QLayout</tt> has some subclasses that can be used to logically arrange things. A favorite is to us <tt class="docutils literal">QVBoxLayout</tt> or <tt class="docutils literal">QHBoxLayout</tt> since things kind of mostly resize correctly in Qt.</p>
<p>As I mentioned before, Single Document Interface programs are easy to program. A couple of cons however include the fact that it may be challenging to jam advanced functionality into them. Depending on the design or back end, if your user pulls up too instances, it can choke the system resources.</p>
<p>So what do you do if your interfaces need to be more complicated? Let’s look at a few alternatives.</p>
</div>
<div class="section" id="tabbed-document-interfaces">
<h2>Tabbed Document Interfaces</h2>
<p>Tabbed Document Interfaces are an extension of the Single Document Interface. Earlier when I asked what you might use to build a web browser… Well if you’re on desktop, you probably thought of a tabbed interface. It’s the most common.</p>
<img alt="" src="https://benhoff.net/images/chrome-snap.png" />
<p>I’d recommend tabbed document interfaces for anything more complicated than a single page.</p>
<p>Qt comes has a custom class to handle creating tabbed interfaces. <tt class="docutils literal">QTabWidget</tt> and the <tt class="docutils literal">QTabWidget.addWidget</tt> functionality are good spots to start. You can also make your own custom one looking at <tt class="docutils literal">QTabBar</tt>. I’ll typically label the tabs logically and try to jam the most interesting/useful information in the left-most tabs (following the left to right reading pattern).</p>
<p>I think the biggest con that I can think of off the top of my head for a Tabbed design is the inability to change to a single document interface to refer to a previous tab. Web browsers got rid of this limitation by allowing you to pull a tab into a new window. While I believe this can be done with Qt, it would take a large amount of coding to accomplish.</p>
</div>
<div class="section" id="multiple-document-interfaces">
<h2>Multiple Document Interfaces</h2>
<p>And then multiple document interfaces are just interesting. I haven’t seen many Multiple Document Interfaces out in the wild. The best example I have off the top of my head of a Multiple Document Interface that I’ve used is Solidworks, or other mechanical drafting software. Excel also has a Multiple Document Interface.</p>
<img alt="" src="https://i-msdn.sec.s-msft.com/dynimg/IC6922.gif" />
<p>I’m not sure when the best time to use Multiple Document Interfaces would be. The challenge is that instead of letting the desktop environment control the window management, your program does instead. This can break the natural flow of the user, although this may be mitigated if you develop specifically for one platform (looking at you, Windows).</p>
<p>You can create an <span class="caps">MDI</span> in Qt using the <tt class="docutils literal">QMdiArea</tt> and then adding windows using <tt class="docutils literal">QMdiArea.addSubWindow</tt>.</p>
<p>I’d recommend checking out the <a class="reference external" href="https://en.wikipedia.org/wiki/Multiple_document_interface">wikipedia</a> page on the topic. The <a class="reference external" href="https://msdn.microsoft.com/en-us/library/ms997505.aspx?ranMID=24542&ranEAID=TnL5HPStwNw&ranSiteID=TnL5HPStwNw-L9gN68KGHNTwS1y_SVKSfw&tduid=(0b68db1eaba6ffcc15fac5f2d8ab4540)(256380)(2459594)(TnL5HPStwNw-L9gN68KGHNTwS1y_SVKSfw)()">Micrsoft Developers Network</a> also has some documentation that might be of further use.</p>
</div>
<div class="section" id="mixed">
<h2>Mixed</h2>
<p>As been hinted at several times already, there’s always the option to mix these main interfaces as much as possible. Keep in mind as well, that there is a lot of support for toolbars and dock widgets (check out the <tt class="docutils literal">QMainWindow</tt> documentation). The Integrated Developer Environment (<span class="caps">IDE</span>) <tt class="docutils literal">Spyder</tt> is a great example of using dock widgets to great effect.</p>
<p>As I wrap up, I’d like to recognize this excellent blog post on <a class="reference external" href="https://richnewman.wordpress.com/category/tabbed-document-interface/">User Interface Design for Business Applications</a>. I default to Single Document Interfaces as much as possible until complexity gets me, and then I switch to a tabbed interface. But the aforementioned post helped me ensure I wasn’t overlooking a design while I was crafting my latest creation.</p>
</div>
<div class="section" id="wrap-up">
<h2>Wrap Up</h2>
<p>Now that we know a couple of different design patterns, let’s go ahead and apply them! Which is exactly what we’ll do in the next section of where we look at <a class="reference external" href="https://benhoff.net/pyqt-layout-design.rst">layout management in PyQt</a>. Or, if you’d like to jump around, <a class="reference external" href="https://benhoff.net/pyqt-tutorial.rst">go to the top level index</a> and jump to the section of the tutorial that interests you most.</p>
</div>
Types of Programs2017-06-17T22:01:00-05:002017-06-17T22:01:00-05:00Ben Hofftag:benhoff.net,2017-06-17:/types-of-programs.html<p class="first last">Experienced programmers know one thing beginner programmers don’t: types of programs.</p>
<p>I’ve had a lot of people ask my how to learn how to program.</p>
<p>And my favorite analogy for that, is that learning programming is a lot like learning carpentry. No one really asks how you become a carpenter. They just go off and build things. Tables, chairs, shelves, boxes.</p>
<p>Programming is the same way. You should just go off and build things. However, you’re not sitting on a desktop application right now, or eating off a command line tool everyday. Programming doesn’t have simple real life analogies that you can figure out how to build. And it’s also hard to see how programming things are constructed.</p>
<p>So when you tell people to go off and build things, they’re not going to know what to build. How do you Google something that you don’t even know anything about?</p>
<p>So during this post, we’re going to go over some of the major client-side types of applications you can build. Or to go back to the earlier carpenter analogy… we’re going to talk about some of the furniture that you can build that you interact with on a daily basis. The programming desks and chairs, if you will.</p>
<p>Up front disclaimer, the libraries I’m going to discuss are all Python libraries. However, if you’re programming in any other language, you should be able to Google <tt class="docutils literal">java tui library</tt> for example and come up with similar libraries in your preferred language.</p>
<div class="contents topic" id="contents">
<p class="topic-title first">Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#command-line-interfaces" id="id1">Command Line Interfaces</a></li>
<li><a class="reference internal" href="#terminal-user-interfaces" id="id2">Terminal User Interfaces</a></li>
<li><a class="reference internal" href="#graphical-user-interfaces" id="id3">Graphical User Interfaces</a></li>
<li><a class="reference internal" href="#web-pages" id="id4">Web Pages</a></li>
<li><a class="reference internal" href="#knowledge-application" id="id5">Knowledge Application</a></li>
</ul>
</div>
<div class="section" id="command-line-interfaces">
<h2><a class="toc-backref" href="#id1">Command Line Interfaces</a></h2>
<p>First and most simple is the command line interface (<span class="caps">CLI</span>). Love or hate them, a lot things in the programming world are <span class="caps">CLI</span>’s. These programs are run from the shell and usually have different arguments passed in. Very popular in system administration due to the lack of graphical user interfaces installed in servers.</p>
<p>Python is a great example of a command line interface program.</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> python runfile.py
</pre></div>
<p>In this case, <tt class="docutils literal">python</tt> is the program being run, while <tt class="docutils literal">runfile.py</tt> is the argument being passed in.</p>
<p>I often use the <tt class="docutils literal"><span class="pre">-m</span></tt> module flag in order to make virtual environments for example.</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> python -m venv myvenv
</pre></div>
<p><tt class="docutils literal">python</tt> is again the program being run, the <tt class="docutils literal"><span class="pre">-m</span></tt> “module” flag tells python that I want to run a particular python module (the <tt class="docutils literal">venv</tt> module in this case), and the <tt class="docutils literal">venv</tt> module takes another argument to specify what I want the directory for the virtual environment to be named (<tt class="docutils literal">myvenv</tt> in this case).</p>
<p>One of the more advanced command line (non-programming-related) programs I’ve used is <a class="reference external" href="https://github.com/rg3/youtube-dl">youtube-dl</a>.</p>
<p>The basic syntax for calling this program goes something like this</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> youtube-dl <span class="o">[</span>OPTIONS<span class="o">]</span> URL <span class="o">[</span>URL...<span class="o">]</span>
</pre></div>
<p>Several libraries in python exist to help you program a <span class="caps">CLI</span>.</p>
<p>Probably the most beginner friendly one is <a class="reference external" href="http://click.pocoo.org/5/">click</a>. However Python comes bundled with two in the standard library, including <a class="reference external" href="https://docs.python.org/3/library/argparse.html">argparse</a> and <a class="reference external" href="https://docs.python.org/3/library/getopt.html">getopt</a>. I tend to use <tt class="docutils literal">argparse</tt> in my personal projects. Your mileage may vary.</p>
<p><span class="caps">CLI</span>’s are common for scripting programs.</p>
</div>
<div class="section" id="terminal-user-interfaces">
<h2><a class="toc-backref" href="#id2">Terminal User Interfaces</a></h2>
<p>Terminal User Interfaces (TUIs) really only got named that way after Graphical User Interfaces (GUIs) came into existence. These programs are run in the terminal and tend to take over and redraw the entire terminal. According to Wikipedia, TUIs display computer graphics in text form. I just tend to think of them as programs that can be run from, and take over, the terminal.</p>
<p>Probably the best know TUIs are famous text editors, including Vim and <span class="caps">EMACS</span>.</p>
<img alt="" src="https://upload.wikimedia.org/wikipedia/commons/8/8c/Vim-%28logiciel%29-console.png" style="width: 546px; height: 324px;" />
<p>Again, there are several libraries to help implement TUIs. The <a class="reference external" href="https://docs.python.org/3/library/curses.html">curses</a> library is probably the most well know library to create a <span class="caps">TUI</span>, although it is only available in linux. <a class="reference external" href="https://github.com/urwid/urwid">urwid</a> is a bit more fully featured library for <span class="caps">TUI</span> creation.</p>
<p><span class="caps">TUI</span>’s are probably less common, although I’ve been interested in creating some for my own personal projects (not enough free time to grok the development process with <tt class="docutils literal">urwid</tt>). All of the goodness of a <span class="caps">CLI</span>, as you’re still in the terminal, with a little bit more ease of use (man pages or help only get’s you so far).</p>
</div>
<div class="section" id="graphical-user-interfaces">
<h2><a class="toc-backref" href="#id3">Graphical User Interfaces</a></h2>
<p>According to Wikipedia, Graphical User Interfaces (GUIs) are a type of interface that allows users to interact with electronic devices through graphical icons and visual indicators. If you figure out what that means, let me know. That aside, <span class="caps">GUI</span>’s are the bread and butter of computing. Your web browser is a <span class="caps">GUI</span>. Your games run in GUIs. If you’ve ever used the Windows or Mac Operating Systems, the predominate way you interact with the computer is through the use of <span class="caps">GUI</span>’s.</p>
<div class="figure">
<img alt="" src="https://upload.wikimedia.org/wikipedia/en/5/54/Microsoft_Office_2016_Screenshots.png" style="width: 546px; height: 360px;" />
<p class="caption">^ All GUIs.</p>
</div>
<p>Python has a host of libraries for building GUIs. There’s the <a class="reference external" href="https://www.gtk.org/A"><span class="caps">GTK</span></a> library. My personal favorite, the <a class="reference external" href="https://www.qt.io/">Qt</a> framework (especially the <a class="reference external" href="https://riverbankcomputing.com/software/pyqt/intro">PyQt</a> bindings of the Qt framework). The Python standard library also has <a class="reference external" href="https://docs.python.org/3/library/tk.html">tkinter</a>.</p>
<p>GUIs are used to build just about anything, and most normal people would look at you funny if you told them a program needed to be run from the terminal. The only thing that has surpassed them in popularity has been…</p>
</div>
<div class="section" id="web-pages">
<h2><a class="toc-backref" href="#id4">Web Pages</a></h2>
<p>You could argue this one, but the truth is a lot of applications today are created to be served as a web page. The web page has become a standard user interface paradigm.</p>
<p>The standard stack for a client side webpage uses html (words and format), <span class="caps">CSS</span> (styling), and JavaScript (dynamic scripting).</p>
<p>If you want to build something programmatic on a web page, you need to program in JavaScript. Or at least for the client (user facing) side. On the backside/server-side… Different story.</p>
<p><a class="reference external" href="http://flask.pocoo.org/">Flask</a> or <a class="reference external" href="https://www.djangoproject.com/">django</a> are two web frameworks for python. <tt class="docutils literal">Django</tt> is a bit more full featured with database model helper classes built in, while <tt class="docutils literal">flask</tt>, as a micro framework, is much less opinionated.</p>
<p>I would recommend looking into a static site generator such as <a class="reference external" href="https://blog.getpelican.com/">pelican</a> if you’re looking at creating something that just needs to be read.</p>
<p><a class="reference external" href="http://jupyter.org/">Jupyter notebooks</a> are also a great way to deliver content via the web.</p>
<p>Web pages are a very common way to deliver an application experience. They are arguably the most cross-platform and consistent user experience.</p>
</div>
<div class="section" id="knowledge-application">
<h2><a class="toc-backref" href="#id5">Knowledge Application</a></h2>
<p>So how does this information help you? We’ve covered the major types of client-side interfaces. If you wanted some inspiration for projects, but want to minimize the user interface portions of your programming, I might suggest researching existing <span class="caps">CLI</span>’s that sound interesting or implement your own. Want something a little bit more discoverable and easy to navigate, but still in the terminal (maybe a low usage application)? TUIs, my friend. Need to create something that others will use on the computer? Maybe get into some <span class="caps">GUI</span> programming. Bowing down to the overwhelming web pressure? Well go learn some JavaScript for your client-side needs. But if it’s a back-end you need, maybe <tt class="docutils literal">flask</tt> or <tt class="docutils literal">django</tt> can save you.</p>
<p>The point is to help provide a little bit of information into the terminology and types of programming applications, so that when you want to make your programming chair, you’re a little more knowledgable about where to go searching.</p>
</div>
PyQt Hello World Application2017-06-16T13:27:00-05:002017-06-16T13:27:00-05:00Ben Hofftag:benhoff.net,2017-06-16:/pyqt-hello-world-application.html<p class="first last">Creating the hello world app of desktop applications.</p>
<div class="section" id="getting-started">
<h2>Getting Started</h2>
<p>This is the first step in the <a class="reference external" href="https://benhoff.net/pyqt-tutorial.rst">PyQt Tutorial series</a>! Welcome!</p>
<p>We’re going to start with a hello world application, as sometimes getting everything installed and working correctly can be an emotionally terrifying experience. Just ask me. I’ve done software before.</p>
<p>So starting from nothing.</p>
</div>
<div class="section" id="install-python">
<h2>Install Python</h2>
<p><a class="reference external" href="http://python-guide-pt-br.readthedocs.io/en/latest/starting/installation/">Follow these installation instructions</a>, based on your <span class="caps">OS</span>.</p>
<p>If you’re running a linux distribution, you can likely use your native package manager.</p>
<p>Make sure if you’re running Windows that you add Python to your path library. It should be a pop-up option during the install. If you mess up, <cite>turn to Google</cite>.</p>
<p>Make sure you install Python 3. This tutorial will not be for <a class="reference external" href="https://pythonclock.org/">Python 2</a> as Python 2 is reaching it’s end of life for official support.</p>
</div>
<div class="section" id="create-a-working-directory">
<h2>Create A Working Directory</h2>
<p>Create a directory you want to work out of.</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> mkdir pyqt
<span class="gp">$</span> <span class="nb">cd</span> pyqt
</pre></div>
</div>
<div class="section" id="create-virtual-environment">
<h2>Create Virtual Environment</h2>
<p>We’re going to create and activate a virtual environment, so that your global site-packages stay clean. Hard to keep track of which project, installed which dependencies, otherwise.</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> python -m venv <DIR>
<span class="gp">$</span> <span class="nb">source</span> <DIR>/bin/activate
</pre></div>
</div>
<div class="section" id="install-pyqt5">
<h2>Install PyQt5</h2>
<p>Right, now that our virtual environment is activated, let’s install PyQt.</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> pip install PyQt5
</pre></div>
</div>
<div class="section" id="build-the-framework">
<h2>Build the Framework</h2>
<p>Now, fire up your favorite text editor, it’s time to start programming. Create a file named <cite>__main__.py</cite> and add the following to it.</p>
<div class="highlight"><pre><span></span><span class="c1"># file name: __main__.py</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="k">pass</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">main</span><span class="p">()</span>
</pre></div>
</div>
<div class="section" id="write-pyqt-specific-code">
<h2>Write PyQt Specific Code</h2>
<p>Now, in our main function, we’re going to create an instances of <a class="reference external" href="http://doc.qt.io/qt-5/qapplication.html">QApplication</a> and <a class="reference external" href="http://doc.qt.io/qt-5/qmainwindow.html">QMainWindow</a>. I’d recommend reading the <a class="reference external" href="http://doc.qt.io/qt-5/qapplication.html#details">documentation</a> on <tt class="docutils literal">QApplication</tt>, especially the section with the <em>main areas of responsibility</em>.</p>
<p>If you don’t get there, the <span class="caps">TLDR</span> on <tt class="docutils literal">QApplication</tt> responsibilities:</p>
<ul class="simple">
<li>event handling</li>
<li>main settings</li>
</ul>
<p><tt class="docutils literal">QMainWindow</tt> is a magical class. It gives a lot of normal desktop interface aspects that you’ve come to expect, such as a menu bar, tool bars, and status bars… <em>for free</em>. Again, the <a class="reference external" href="http://doc.qt.io/qt-5/qmainwindow.html#details">documentation is awesome</a>, but check out the below figure to get a hint for the types of functionality that are integrated into the <tt class="docutils literal">QMainWindow</tt> class.</p>
<div class="figure align-center">
<img alt="" src="http://doc.qt.io/qt-5/images/mainwindowlayout.png" />
<p class="caption">Notice that the Menu Bar and Status Bar are tied to QMainWindow. We’ll come back to that.</p>
</div>
<p>So now that we now a little bit about what we’re doing, let’s write the rest of the code. The entire source script is available <a class="reference external" href="https://github.com/benhoff/blog/blob/master/scripts/pyqt-hello-world.py">at this link</a> as well.</p>
<div class="highlight"><pre><span></span><span class="c1"># file name: __main__.py</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">from</span> <span class="nn">PyQt5</span> <span class="kn">import</span> <span class="n">QtCore</span><span class="p">,</span> <span class="n">QtWidgets</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="c1"># We need to make the QApplication before our QMainWindow</span>
<span class="c1"># We also need to pass in our system argument values (sys.argv)</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">QtWidgets</span><span class="o">.</span><span class="n">QApplication</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span>
<span class="n">main_window</span> <span class="o">=</span> <span class="n">QtWidgets</span><span class="o">.</span><span class="n">QMainWindow</span><span class="p">()</span>
<span class="c1"># QMainWindow requires a central widget, so we'll just pass in a</span>
<span class="c1"># blank widget for now</span>
<span class="n">blank_widget</span> <span class="o">=</span> <span class="n">QtWidgets</span><span class="o">.</span><span class="n">QWidget</span><span class="p">()</span>
<span class="n">main_window</span><span class="o">.</span><span class="n">setCentralWidget</span><span class="p">(</span><span class="n">blank_widget</span><span class="p">)</span>
<span class="c1"># Show our main window</span>
<span class="n">main_window</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="c1"># Start the event loop processing</span>
<span class="n">app</span><span class="o">.</span><span class="k">exec</span><span class="p">()</span>
</pre></div>
<!-- TODO Get photo evidence -->
<p>That’s it! You should see a small window open up. We’ve got the “Hello World” version for <span class="caps">GUI</span> programming.</p>
<p>The next step is to talk about <a class="reference external" href="https://benhoff.net/qt-interface-design.rst">choices for our central widget design</a>, but if you’re an advanced PyQt programmer already, you can checkout <a class="reference external" href="https://benhoff.net/pyqt-tutorial.rst">this high level overview</a> of the tutorial series to jump around to the topic you need.</p>
</div>
Software Engineering2017-06-15T09:05:00-05:002017-06-15T09:05:00-05:00Ben Hofftag:benhoff.net,2017-06-15:/on-software-engineering.html<p class="first last">Hiring differences between being a Software Engineer and Computer Scientist</p>
<p>The other day, I ran into the idea of <a class="reference external" href="http://sijinjoseph.com/programmer-competency-matrix/">The Programmer Competency Matrix</a>. Which as I scrolled through the various categories and read the various rows I realized: Wow. I am not very competent.</p>
<p>Except for Software Engineering.</p>
<p>I am level 3 in all categories of Software Engineering except for testing, where I’m a solid two (<span class="caps">UI</span> testing would be new to me). I would probably be a Level 1 in all Computer Science categories.</p>
<p>And I’m completely <span class="caps">OK</span> with this.</p>
<p>I’ve had a lot of folks ask me while I was <a class="reference external" href="https://www.youtube.com/channel/UChWbNrHQHvKK6paclLp7WYw">live streaming</a> heavily how to get better at programming. And if this matrix is any indication, I’m the wrong person to ask.</p>
<p>Now I think the question they were really asking is how do I feel more comfortable navigating my programming language’s <span class="caps">API</span> or be better at integrating third party libraries…</p>
<p>But <em>if</em> the question they were asking is how do I better prepare myself to get hired, I would respond with the following:</p>
<p><strong>Software Engineering</strong></p>
<p>As a (now) <em>working</em> Mechanical Engineer, very little of my day job requires engineering. I’m working in supplying spare parts, which is the perfect physical analogy to maintaining existing systems. My only job is to not break it and keep it running. The design engineering has already been done.</p>
<p>I would argue that most software jobs are similar. Keep the system running and don’t break it.</p>
<p>And when you’re trying to do that, documentation and process become the most important steps. What is this? Why did it break? When I fix it, will someone else understand what I did? Additional software engineering terms, how do I commit changes? Who approves them? What testing needs to be done? Or types of tests written?</p>
<p>These types of questions will take up more than binomial or Fibonacci heaps [data structures, level 3].</p>
<p>What do you think about the debate of Software Engineering vs Computer Scientist? Let me know in the comments below.</p>
Flashing Beaglebone Black2017-06-12T19:51:00-05:002017-06-12T19:51:00-05:00Ben Hofftag:benhoff.net,2017-06-12:/flashing-beaglebone-black.html<p class="first last">Trails and tribulations of trying to bring a Beaglebone black up to the latest kernel.</p>
<p>All right, let’s update a <a class="reference external" href="https://beagleboard.org/black">beaglebone black</a>.</p>
<p>I need to flash the thing.</p>
<p>So <a class="reference external" href="https://beagleboard.org/latest-images">downloading</a> the latest image from the repository, I need to unpack the image.</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> unxz bone-debian-X.X-iot-armhf-YYYY-MM-DD-4gb.img.xz
</pre></div>
<p>Ok, now I’ve got the uncompressed image.</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> lsblk
</pre></div>
<p>Shows the <span class="caps">SD</span> card I’ve just plugged in as /dev/sdc</p>
<div class="highlight"><pre><span></span><span class="go">sudo mount /dev/sdc /mnt</span>
</pre></div>
<p>Then let’s write our image.</p>
<div class="highlight"><pre><span></span><span class="go">sudo dd if=bone-debian*.img of=/dev/sdc bs=512 status=progress</span>
</pre></div>
<p>Believe me, that little ‘status=progress’ bit is a life saver. Otherwise you have no idea how long it’s going to take. I also don’t know how to kill dd without pulling out the drive.</p>
<p>The last two sentences are related.</p>
<p>I’m following <a class="reference external" href="http://derekmolloy.ie/write-a-new-image-to-the-beaglebone-black/">this</a> blog post by the way.
Since you’re going to be awhile.
My image size was 4 gigabytes with a transfer speed of ~6 megabytes/second.</p>
<p>And it died. 13 minutes of waiting… Ok. Let’s try again. (<a class="reference external" href="http://guichaz.free.fr/iotop/">Iotop</a> will show you if the copy is still working)</p>
<p>Died again. 3rd times the charm.
Nope. Ok on to something else. Normally I’d just go get a new <span class="caps">SD</span> card, but I’m transferring this through like 3 adapters and it’s been known to give me issues in the past.</p>
<p>Stack Exchange to the rescue. According to <a class="reference external" href="https://unix.stackexchange.com/questions/180330/resuming-a-dd-of-an-entire-disk">this post</a> I should just be able to resume where I left off. Neat.</p>
<p>You’ll want to seek and skip. The output of should look something like this.</p>
<div class="highlight"><pre><span></span><span class="go">6963200+0 records in</span>
<span class="go">6963200+0 records out</span>
<span class="go">3565158400 bytes (3.6 GB, 3.3 GiB) copied, 732.023 s, 4.9 MB/s</span>
</pre></div>
<p>You’ll want to use the records for the seek and skip variables referenced above.</p>
<p>However, after some serious heartache, it looks like my dd just didn’t terminate correctly, and everything was written.</p>
<p>So now, holding down the boot button, waiting for the buttons to start flashing the heartbeat pattern, and it looks like we’re in business.</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> ssh debian@192.168.7.2
</pre></div>
<p>The password was in the banner for me.</p>
<p>Now on to <a class="reference external" href="http://kacangbawang.com/beagleboneblack-revc-debloat-part-1/">demucking</a> the install.</p>
Mailing List Woes2017-06-12T16:01:00-05:002017-06-12T16:01:00-05:00Ben Hofftag:benhoff.net,2017-06-12:/mailing-list-woes.html<p class="first last">Figuring out how to add a mailing list.</p>
<p>All right, let’s add a mailing list.</p>
<p>So I know I’ve got <a class="reference external" href="https://www.mailgun.com">Mailgun</a> use for <a class="reference external" href="http://benhoff.net/adding-discourse-comments.html">Discourse</a> (used for forum comments). Looking into the Mailgun documentation, I can see that there’s an actual <a class="reference external" href="https://documentation.mailgun.com/en/latest/api-mailinglists.html"><span class="caps">API</span></a> for a mailing list. However the documentation shows Mailgun’s implementation is more of a backend deal. Which would be fine if I had a dynamic application, but since I’m dealing with static pages, this is a major hangup.</p>
<p>So my next option is to search for a look for either an already existing solution or host my own.</p>
<p>This is a mailing list. I could build one with sqlite and python in like 5 minutes. And it would be awful.</p>
<p>Let’s start by looking to see what I can host myself.</p>
<p>After some google searching, it looks like the only open source version is <a class="reference external" href="https://www.gnu.org/software/mailman/">Mailman</a>.</p>
<p>I’ve got a confession to make. I’ve tried to get Mailman up and running before. It didn’t go very well. Mailman is a mailing list manager. It provides a way to see past emails, navigate replies, and search. Mailman does not provide a way to actually send emails. You have to hook either Postfix or something else up to actually get it up and running.</p>
<p>Also the other thing is, I’ve already got a full up forum with Discord. And right now, that forum is consuming 1 of 1 gigabytes of data that the Digital Ocean instance provides me. And I don’t want to upsize the instance.</p>
<p>So now I’m back to using a hosted service. A couple of blogs recommended using <a class="reference external" href="https://www.aweber.com/">Aweber</a>, who I’ve never heard of, but I’m going to stick with <a class="reference external" href="https://mailchimp.com/">Mailchimp</a>, mostly because I’ve heard of them actually.</p>
<p>So signed up, started building a mailing list, got the html code to put into the site’s template. Which is perfect. I’ll eventually work towards sending out weekly “round-up” emails. Just a simple email that lays out all the articles so you don’t have to visit the site to see what I’m writing about.</p>
<p>But there’s some pieces I need to figure out. Since I’ve got my own domain and a transactional mailing agent with Mailgun, I should be able to actually receive emails at the domain. As in: <a class="reference external" href="mailto:ben@benhoff.net">ben@benhoff.net</a></p>
<p>I set up a few referral rules at Mailgun, but it’s a little dicey currently. I’ve gotten a few emails, but not all of them. I’m not sure what’s wrong, but I think it has something to do with the <span class="caps">DNS</span> records.</p>
<p>If you have ever experienced or might now what’s going on, feel free to comment below and point me in some resources. Otherwise, I’ll dive into it later and post another article detailing the fix.</p>
Blog Tech Stack2017-06-12T08:34:00-05:002017-06-12T08:34:00-05:00Ben Hofftag:benhoff.net,2017-06-12:/blog-tech-stack.html<p class="first last">Reviewing the tech stack and hosting services I’m using for the blog.</p>
<p>As I’m looking to start up a mailing service I thought it be interesting to talk about the services I’m using for hosting this blog. I’ll do it in chronological order. Kind of.</p>
<table border="1" class="docutils">
<colgroup>
<col width="35%" />
<col width="65%" />
</colgroup>
<thead valign="bottom">
<tr><th class="head">Service Type</th>
<th class="head">Software/Provider</th>
</tr>
</thead>
<tbody valign="top">
<tr><td>Static Site Generator</td>
<td><a class="reference external" href="https://blog.getpelican.com/">Pelcian</a></td>
</tr>
<tr><td>Static Site Host</td>
<td><a class="reference external" href="https://pages.github.com/">Github Pages</a></td>
</tr>
<tr><td>Comments/Forum</td>
<td><a class="reference external" href="https://www.discourse.org/">Discourse</a></td>
</tr>
<tr><td>Dynamic Content Host <a class="footnote-reference" href="#id3" id="id1">[1]</a></td>
<td><a class="reference external" href="https://m.do.co/c/2fdf30b46683">Digital Ocean</a></td>
</tr>
<tr><td>Domain Registration/<span class="caps">DNS</span></td>
<td><a class="reference external" href="https://domains.google/#/">Google Domains</a></td>
</tr>
<tr><td>Transactional Email <a class="footnote-reference" href="#id4" id="id2">[2]</a></td>
<td><a class="reference external" href="https://www.mailgun.com/">Mailgun</a></td>
</tr>
<tr><td>Mailing List</td>
<td><a class="reference external" href="https://mailchimp.com/">Mailchimp</a></td>
</tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id3" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>Used for hosting the discourse instance</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id4" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>Transactional email is, according to <a class="reference external" href="https://blog.mailchimp.com/what-is-transactional-email/">Mailchimp’s Docs</a>, an email sent to an individual based on some action. Discourse uses it for signups, which is why I needed it.</td></tr>
</tbody>
</table>
Adding Discourse Comments2017-06-10T14:25:00-05:002017-06-10T14:25:00-05:00Ben Hofftag:benhoff.net,2017-06-10:/adding-discourse-comments.html<p class="first last">Working through adding Discourse Comments to the blog.</p>
<p>Alright, here we go. Let’s add comments to my static site. I waited until the weekend because I knew it was going to be a pain.</p>
<p>So following the directions, I created a digital ocean droplet, and copied the github repo over.</p>
<p>This was relatively easy for me. Had some issue with path wrangling, but got it figured out eventually. Github wants to ask for your github username weirdly when doing a git clone if you do it incorrectly. Eventually I just changed into the directory I wanted the repo and did the clone from there. Feel like I shouldn’t have to do that, but whatever.</p>
<p>Ok, so now I need email. Damn.</p>
<p>Well let’s stick with mailgun, since that sounds familiar.</p>
<p>So I signed up for mailgun, but now I need a domain and a <span class="caps">DNS</span> provider.</p>
<p>So I signed up for a domain on Google. <cite>benhoff.net</cite>. Old school. Perfect.</p>
<p>Just remembered that a domain and a <span class="caps">DNS</span> provider are different. Wonder if google will act as a <span class="caps">DNS</span> provider. I’m swimming in unknown territory now.</p>
<p>Ok, I need to add <span class="caps">DNS</span> records for sending according to mailgun.</p>
<p>Uhh, I don’t know what I’m doing. Luckily there’s this blogpost. <a class="reference external" href="http://www.curtismlarson.com/blog/2015/04/12/github-pages-google-domains/">http://www.curtismlarson.com/blog/2015/04/12/github-pages-google-domains/</a>
What an annoying pop up page.</p>
<p>Ok, I followed the directions to add the custom resource records from my google name to my google domain. According to the blog, it should take about a day to resolve.</p>
<p>I don’t think this solves my mailgun issue though.</p>
<p>A little google fu: <a class="reference external" href="https://stackoverflow.com/questions/37864807/setting-up-google-domains-dns-to-work-with-mailgun-mx-records">https://stackoverflow.com/questions/37864807/setting-up-google-domains-dns-to-work-with-mailgun-mx-records</a></p>
<p>Looks like I need to add a subdomain as a <span class="caps">MX</span> record.</p>
<p>So under Custom Resource records, I’m going to add the name <cite>mg</cite>, with the type <cite><span class="caps">MX</span></cite> and add the <cite>mxa.mailgun.org</cite> and <cite>mxb.mailgun.org</cite>. Looks like google automagically adds a 10 and a period to the end of that.</p>
<p>Cool, so now that I’ve done that, I’ve added the domain <cite>mg.benhoff.net</cite> to the mailgun side.</p>
<p>Now I need to add everything else.</p>
<p>From this site: <a class="reference external" href="http://code.krister.ee/mailgun-digitalocean/">http://code.krister.ee/mailgun-digitalocean/</a></p>
<p>Looks like I just keep adding these to the custom resource records.</p>
<p>I added email.mg <span class="caps">CNAME</span> to point to mailgun.org, and a krs._domainkey.mg to point to a really long string. By clicking the authorize, looks like everything worked.</p>
<p>All right, back to the discourse setup.</p>
<p>I’ll launch the setup tool. Setup wants to create a 2 gig swap file. Fine by me. Following the defaults for mailgun set by: <a class="reference external" href="https://github.com/discourse/discourse/blob/master/docs/INSTALL-email.md">https://github.com/discourse/discourse/blob/master/docs/<span class="caps">INSTALL</span>-email.md</a></p>
<p>Now it asks for let’s encrypt account email? Hmm, let’s set that up real quick.</p>
<p>Just enter an email and press enter, and everything will be encrypted.</p>
<p>Discourse is now updating which is fine.</p>
<p>I lost internet, but looks like everything worked fine. Perfect.</p>
<p>I’ll add the <span class="caps">CNAME</span> to the custom resource records. Whoops, looks that’s not how to do that.</p>
<p>I need to add it as a ‘A’ resource and fix the domain name in my setup and rebuild the app using <cite>./launch rebuild app</cite></p>
<p>Sick. I’ll add in some other fancy stuff, like Facebook, Google, Twitter, and Github logins.</p>
<p>Now onto embedding. I’m following this: <a class="reference external" href="https://meta.discourse.org/t/embedding-discourse-comments-via-javascript/31963">https://meta.discourse.org/t/embedding-discourse-comments-via-javascript/31963</a></p>
<p>The admin tab has a customize section. From there to embedding. Add the domain name and I’ve got some html. Adding it to the theme of the article template and I should be set.</p>
<p>Fingers crossed.</p>
Fixing My Issue With Github Stars2017-06-09T08:11:00-05:002017-06-09T08:11:00-05:00Ben Hofftag:benhoff.net,2017-06-09:/remembering-github-stars.html<p class="first last">How I fixed a long term issue with how I star repositories on Github</p>
<p>I’ve got this weird problem.</p>
<p>I don’t remember why I’ve starred the things I’ve starred on Github. It’s gotten so bad in fact, that I’ll often be browsing an interesting Github repo on a topic I’m interested in, go to bookmark it, and realize that I already have.</p>
<p>Doh!</p>
<p>The real issue is that the reason I bookmark repo’s is for a very specific purpose. For example, a lot of the bot code that I follow is because I’m interested in how different bots parse and handle incoming messages or events.</p>
<p>I mean, in my head, it’s probably something as simple as pushing it onto a queue and then using a worker thread to work through the tasks. But what if it’s different?</p>
<p>Part of the reason I started working on Vexbot was because I was interested in a universal interface to anything. I wanted to take and plug into random services (Youtube, Irc, etc) and have them all run through a common parsing/bot system. So far, I’ve created the only bot that I’ve seen that uses subproccesses for each individual adapter.</p>
<p>But I’m always curious about what others have done to tackle this problem, so I bookmark other bots to go check out how they do it.
Except I never do. And then browsing through my list of stars (which I rarely do) I can never remember why I bookmarked it.</p>
<p>So let’s fix it.</p>
<p>I’ll create a github repo targeted at remembering why I starred the things I stared. By using a Readme file, I’ll only have to visit the main repo page. Plus very thing is version controlled and nice. Something that would be nice would be to create a bot that periodically polls my starred list and creates an issue or a pull request so that I don’t get lazy, but let’s not get ahead of ourselves.</p>
<p>So hold my beer. Here I go.</p>
Bash String Subsitution2017-06-08T13:19:00-05:002017-06-08T13:19:00-05:00Ben Hofftag:benhoff.net,2017-06-08:/bash-string-substitution.html<p class="first last">Learning how to do string substitution in bash commands</p>
<p>A little personal confession: I’m really bad at the command line. And by bad, I mean willfully ignorant. I often know there’s a better way to work in the command line and I often don’t learn how to do it.</p>
<p>For example, one of the little tricks that has saved me a lot of time is the use of the double exclamation marks to redo and edit a command.</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> pacman -S interesting-package
<span class="go">error: you cannot perform this operation unless you are root</span>
<span class="gp">$</span> sudo !!
<span class="go">sudo pacman -S interesting-package</span>
</pre></div>
<p>Using the double exclamation marks will save you from retyping all of the things when you forget something simple like a sudo. It’s saved my sanity many a times.</p>
<p>One example of my willful ignorance, is with misspelling things on the command line</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> git comit -m <span class="s1">'really long commit message'</span>
<span class="go">git: 'comit' is not a git command. See 'git --help'</span>
<span class="go">The most similar command is</span>
<span class="go"> commit</span>
</pre></div>
<p>Dammit. Up until today, I’ve pressed the up arrow to bring the command back up, then used the left arrow to navigate all the way back to my mistake before fixing it.</p>
<p>The crazy thing is I know there’s a way to do a simple string substitution. I’ve just been too lazy to look it up.</p>
<p>Until today. Today, with the use of Google, I become a power user.</p>
<p>As a vim user this is the syntax that I’ll remember.</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> !!:s/comit/commit
</pre></div>
<p>Alternatively, if you’ve invested in a saner text editor, a simpler syntax is:</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> ^comit^commit^
</pre></div>
<p>Now, if only I could remember the way to navigate to the beginning of a command line (<cite>Ctl-e?</cite>…. <cite>Ctl-a?</cite>)</p>
<p>or learn how to use my bash history….</p>
<p>Well there’s always tomorrow.</p>
Blog Implementation Stream of Conscious2017-06-04T21:27:00-05:002017-06-04T21:27:00-05:00Ben Hofftag:benhoff.net,2017-06-04:/blog-implementation.html<p class="first last">Starting implementation of the blog as a work in progress.</p>
<p>All right, let’s start implementing a blog.</p>
<p>First off, let’s get the Github Pages up first.</p>
<p>Going to <a class="reference external" href="https://pages.github.com/">Github Pages site</a> gives the instructions.</p>
<p>I need to create a repository named after my username and push the repo. Easy enough. Next I need to create an index.html file. Github recommends “hello world”. Sounds good to me.</p>
<p>Using git to add and push my local repo up to the newly created repository. Visiting benhoff.github.io now shows our the new web page, hello world! Neat!</p>
<p>Cool, now that I’ve got my page up, on to the static website generation. I’ll start by installing pelican in a virtual environment.</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> mkdir swdev/blog
<span class="gp">$</span> <span class="nb">cd</span> swdev/blog
<span class="gp">$</span> python -m venv venv
<span class="gp">$</span> <span class="nb">source</span> venv bin activate
<span class="gp">(venv) $</span> pip install pelican
</pre></div>
<p>Pelican has a quick start. Hold my breath here ;)</p>
<div class="highlight"><pre><span></span><span class="gp">(venv) $</span> pelican-quickstart
</pre></div>
<p>Oh man, I need a blog title? Hoff’s heroes? Let’s just stick with Ben’s Blog for now. The only other question I’m stumped on is the generation of a Fabfile/Makefile. Let’s say yes for now, and we can always delete it later.</p>
<p>Looks like pelican is familiar with using Github pages. Nice.</p>
<p>Awesome, the quickstart dumped a bunch of new files and folders into the directory. I’ve already written an initial thoughts post about this blog. Let’s dump that in the content directory.</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> mv initial_blog_thoughts.txt content/initial_blog_thoughts.rst
</pre></div>
<p>I’m following the content tutorial <a class="reference external" href="http://docs.getpelican.com/en/stable/content.html">here</a>.</p>
<p>There’s a bunch of metadata that pelcian says I’m missing, including the date, tags, category, slug, authors, and summary. Let’s go ahead and add all of those.</p>
<p>I’ll need to go through and add in some of the resturctured text for the post, but I’ll do it later.</p>
<p>Right now, I want to add in some <tt class="docutils literal">pages</tt>. According to Pelican, Pages are for _About_ and _Contact_ pages. That sounds good.</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> mkdir pages
</pre></div>
<p>Uh, I thought I could create an <tt class="docutils literal">index.html</tt>? Doesn’t look like I can, I’ll just follow the next step in the tutorial and see where I end up.</p>
<div class="highlight"><pre><span></span><span class="gp">(venv) $</span> pelican /path/to/my/content
</pre></div>
<p>Looks like the content needs to be the file directory and not the actual content.</p>
<p>Nice, the output directory has an index file in it already. Let’s check it out using my browser.</p>
<p>Ok, I’m not in love with the theme, but I can work with everything else.</p>
<p>Let’s push this up and do it live.</p>
<p>Ok, not as easy as I would like. We’re adventuring now. Pelican <a class="reference external" href="http://docs.getpelican.com/en/stable/tips.html">recommends</a> the use of <cite>ghp-import</cite>. I hate this kind of adventuring, since I’m pretty sure it’s not going to work. Sigh. Here we go.</p>
<div class="highlight"><pre><span></span><span class="gp">(venv) $</span> pip install ghp-import
<span class="gp">(venv) $</span> ghp-import output
<span class="gp">(venv) $</span> git push git@github.com:benhoff/benhoff.github.io.git gh-pages:master
</pre></div>
<p>Yea, that didn’t work. Looks like since I’ve already pushed I’m going to have issues. Color me surprised. I swear, this is my surprised face.</p>
<p>I’ve now got a <tt class="docutils literal"><span class="pre">gh-pages</span></tt> branch. That might be worth looking into. After some digging looks like I need to push the gh-pages branch.</p>
<div class="highlight"><pre><span></span><span class="gp">$</span> git push origin gh-pages
</pre></div>
<p>That didn’t appear to work. Maybe deleting my old “hello world” index file will help.</p>
<p>Nope. Now we’re just 404ing. Looks like for a user page, content must be in the master. Well screw that. Let’s create a new repo for the blog code and then I’ll just push the github.io pages separate.</p>
<p><em>New repo named `blog`</em></p>
<div class="highlight"><pre><span></span><span class="gp">$</span> git remote set-url git@github.com:benhoff/blog.git
<span class="gp">$</span> git remote add publish git@github.com:benhoff/benhoff.github.io
<span class="gp">$</span> git push -f publish gh-pages:master
</pre></div>
<p>I’ll still have to manually push the gh-pages branch, but this is good enough for now.</p>
Blog Initial Tech Stack Thoughts2017-06-04T20:30:00-05:002017-06-04T20:35:00-05:00Ben Hofftag:benhoff.net,2017-06-04:/blog-tech-stack-thoughts.html<p class="first last">Walking through the initial tech stack considerations for a new blog.</p>
<p>Ok, let’s start a blog.</p>
<p>First of all, we need a tech stack. My initial thoughts are to use a static site generator. None of the blog code needs to be dynamic, and I can import a javascript comment system (such as discus) later. Probably end up using google analytics as well.</p>
<p>I’ll be using either markdown or restructured text so that I can track the blog configuration using git.</p>
<p>Perfect, so we’ve got a static site generator, markdown/restructured text, third party analytics and commenting… now on to hosting. Since I don’t feel like taking on the hosting thing for now, and the domain name isn’t important, let’s just stick with github pages. I’m pretty sure it’s free.</p>
<p>The only other things that I would like would be some responsive design (I primarily read using my smart phone) and some way to embed a jupyter notebook directly in the page.</p>
<p>I’ll save the responsive design for another day (and a more complicated folder directory as well), but the desire for jupyter notebook means I’ll probably be using some sort of python stack instead of the (likely more popular) Github incarnation of Jekyll.</p>
<p>A quick google search shows that Pelican is likely the largest/most supported python static site generator. Looks like it can work with either markdown or restructured text (I’ll probably stick with restructured text since I’ve written some <span class="caps">README</span>’s with it). Perfect. Another quick search shows there’s a plugin that allows for Jupyter-Notebooks to be in embedded. We’re cooking with fire now. So let’s review our tech.</p>
<ul class="simple">
<li>Git (version control)</li>
<li>Resturctured Text (text markup)</li>
<li>Github Pages (website hosting/Domain)</li>
<li>Pelican (static site generator)</li>
</ul>
<p>Tech to be implemented later (third party and potentially hard)</p>
<ul class="simple">
<li>Discourse (comments)</li>
<li>Google Analytics (analytics)</li>
</ul>
<p>Awesome, let’s start implementing.</p>