Java: Two ways to glob

Two ways of globbing.

First, using a DirectoryStream.  This is the better way.

    static List xglob(String filePath, String wc) {
        List result = new ArrayList();

        Path dir = FileSystems.getDefault().getPath(filePath);
        DirectoryStream stream = null;
        try {
            stream = Files.newDirectoryStream(dir, wc);
            for (Path path : stream) {
                result.add(path.getFileName().toString());
            }
            stream.close();

        } catch (IOException e) {
            // glob of something that does not exist is empty
        }
        return result;
    }

Second, using File().

    static List glob2(String dirname_s, String wc) {
        PathMatcher matcher = FileSystems.getDefault()
                                .getPathMatcher("glob:"+wc);
        List result = new ArrayList();

        File dircontents = new File(dirname_s);
        String[] paths = dircontents.list();

        for (String ps : paths) {
            Path p = Paths.get(ps);
            if (matcher.matches(p)) {
                result.add(p.toString());
            }
        }
        Collections.sort(result);
        return result;
    }

blogodex = {"idx" : ["glob", "java"]};

Mac Installing Consolas font

I thought installing Office would install it in the Font Book.  Apparently not!

Here's how to get it into the Font Book.  Be sure and do this in a scratch directory as it creates a lot of files.

brew install cabextract
curl -O http://download.microsoft.com/download/f/5/a/f5a3df76-d856-4a61-a6bd-722f52a5be26/PowerPointViewer.exe
cabextract PowerPointViewer.exe
cabextract ppviewer.cab
open CONSOLA*.TTF

blogodex = {"idx" : ["Consolas", "fonts", "Mac setup"]};

Saving Youtube Audio for iPhone playback

I'm watching a lot of tutorials these days on Youtube.  A lot of these are fine audio-only, and it would be nice to listen to them on the daily commute.

BUT, unlike the various audio players, the Youtube player pauses when you lock the screen.  So, here's some incantations to grab the audio.

• grab video. I'm using: https://github.com/gantt/downloadyoutube
• ffmpeg -i foo.mp4 -c copy foo.aac
• avconvert  --preset PresetAppleM4VCellular-s foo.aac -o foo.mp3

I think an mp3 codec can be added to ffmpeg.  If anybody knows an easy incantation give me a shout! This may be it:

• ffmpeg -i foo.mp4 -c:a libmp3lame -ac 2 -q:a 2 foo.mp3

blogodex = {"idx" : ["video conversion", "audio extraction", "Youtube"]};

Kafka Training Deck

Covering:

• Introducing Kafka: history, Kafka at LinkedIn, Kafka adoption in the industry, why Kafka
• Kafka core concepts: topics, partitions, replicas, producers, consumers, brokers
• Operating Kafka: architecture, hardware specs, deploying, monitoring, performance and scalability tuning
• Developing Kafka apps: writing to Kafka, reading from Kafka, testing, serialization, compression, example apps (with kafka-storm-starter)
• Playing with Kafka using Wirbelsturm

http://www.michael-noll.com/blog/2014/08/18/apache-kafka-training-deck-and-tutorial/

blogodex = {"idx" : ["kafka", "tutorials"]};

Oracle: quick AS OF demo

First, let's check our current SCN

    scott@orcl> select current_scn from v$database;
    current_scn
    -----------
    7559499070893

And see how many rows we have in table FOO

    scott@orcl> select count(*) from foo;
    count(*)
    --------
          33

Delete everything from FOO

    scott@orcl> delete from foo;
    (33 rows, 0.011 sec)

    scott@orcl> select count(*) from foo;
    count(*)
    --------
           0

What's our SCN now?

    scott@orcl> select current_scn from v$database;
    current_scn
    -----------
    7559499071397

What was the count as of yesterday?

    scott@orcl> select count(*) from foo as of timestamp sysdate-1;
    count(*)
    --------
          33

And as of the SCN at the start of our session?

    (1 rows, 0.010 sec)
    scott@orcl> select count(*) from foo as of scn 7559499070893;
    count(*)
    --------
          33

blogodex = {"toc" : "Oracle SCN", "idx" : ["Oracle", "SCN", "System Change Number", "as of"]};

Installing Maven

Pretty simple:

• Download tarball: https://maven.apache.org/download.cgi
• untar
• sudo mv apache-maven-3.3.0 /opt/maven
• add /opt/maven/bin to your path

This was on a Mac.

blogodex = {"idx" : ["maven", "installation"]};

SQL Developer Keepalive plugin

When you start SQL Developer, click on the icon.  Pressing enter gives you the default of sending a message every 600 seconds.

https://github.com/scristalli/SQL-Developer-4-keepalive

blogodex = {"idx" : {"SQL Developer", "Oracle connection timeout", "ORA-03135"]};

What if your name is actually Null?

Interesting comments by Mr. Christopher Null!

I'm adding "null" to the blogodex, will it come back to bite me?

http://www.wired.com/2015/11/null

blogodex = {"idx" : ["problem areas", "null"]};

Nice article on The Log

"Each working data pipeline is designed like a log; each broken data pipeline is broken in its own way."—Count Leo Tolstoy (translation by Jay Kreps)

Here's an excellent manifesto about "the data log".  You might be familiar with this concept in the guise of database journals, or with event sequences in distributed systems.

Jay starts from there and writes (pretty comprehensively!) about how this idea is used in several modern systems, and talks about his experience at linkedin using this approach.  Lots of interesting references at the end.  I was happy to see John Ousterhout mentioned there!

The Log: What every software engineer should know about real-time data's unifying abstraction

blogodex = {"idx" : ["logging", "Jay Kreps", "Kafka", "fundamentals", "scalability"]};

Some JVM Resources

Three interesting sources:

• JVM Bytecode for Dummies
• Create Your Own Programming Language
• Implement Your Own JVM Compiler

JVM Bytecode for Dummies

Nice presentation by Charles Nutter.

http://www.slideshare.net/CharlesNutter/redev-2010-jvm-bytecode-for-dummies

Monitoring Oracle Dataguard Status

Received a tip from the gurus:

ON STANDBY:
v$dataguard_stats - various stats, including apply lag.
v$dataguard_status - status messages related to dataguard.
v$managed_standby - info on the standby 

ON PRIMARY
v$archive_gap - shows the # of archived redo logs the standby is
behind.  Since we're shipping in real-time, this should nearly always be 0.

Boto 3 Released

Key points:

• works with python 2 and 3
• interoperable with boto
• new package name: boto3
• docs here: https://boto3.readthedocs.org/
• pip install boto3

https://aws.amazon.com/blogs/aws/now-available-aws-sdk-for-python-3-boto3/

Setting a Sphinx theme

To set a theme in Sphinx:

• find a theme you want to copy.  It will generally be a directory full of files.
• copy the directory under your project's _templates directory.
• set the html_theme in conf.py
• some docs mention setting html_theme_path as well, but it doesn't seem necessary if it's under _templates.
• There's some nice themes at https://readthedocs.org

Example:

$ ls _templates/

sphinx_rtd_theme/

$ diff conf.py-orig conf.py

113c113,114

< html_theme = 'alabaster'

---

> html_theme = "sphinx_rtd_theme"

> html_theme_path = ["_themes", ]    ### probably not necessary

"git push" to Github wants me to log in!

If git prompts you for a username when you push:

$ git push

Username: 

Check your remote settings.

$ git remote -v

origin https://github.com/YOURNAME/notebook.git (fetch)

origin https://github.com/YOURNAME/notebook.git (push)

If it's an https reference, you can change it to an ssh reference as follows. 

$ git remote set-url origin git@github.com:YOURNAME/notebook.git

And everything is great!

$ git push

Everything up-to-date

Mathematical Theory of Claude Shannon

Claude Shannon is well-known for having pioneered information theory and switching theory.  Here's a great overview of his approach to these and other areas he studied.

http://web.mit.edu/6.933/www/Fall2001/Shannon1.pdf

Software: Pagegen Static Site Generator

Pagegen converts a directory tree of RST docs into a static site.  "Aims to generate a fully featured site ready for CSS styling," according to the docs.

blogodex = {"toc" : "PageGen", "idx" = ["Software"]};

Book: The Three Body Problem, Liu Cixin

When I was in China, I met some Sci Fi enthusiasts and asked who their favorite authors were.  I wasn't sure who might have been translate, and how the various political points of view might have affected what was brought in to the country.

To my surprise, they had a lot of authors, all of them Chinese.  It turns out that the country's enthusiasm for science led to Science Fiction being very popular in the country.

I hadn't heard of any of them, and none of the works seem to have been translated into English. Perhaps this is going to change, starting with this book?

Amazon link and their blurb:

The Three-Body Problem is the first chance for English-speaking readers to experience this multiple award winning phenomenon from China's most beloved science fiction author, Liu Cixin. 

Set against the backdrop of China's Cultural Revolution, a secret military project sends signals into space to establish contact with aliens. An alien civilization on the brink of destruction captures the signal and plans to invade Earth. Meanwhile, on Earth, different camps start forming, planning to either welcome the superior beings and help them take over a world seen as corrupt, or to fight against the invasion. The result is a science fiction masterpiece of enormous scope and vision.

blogodex = {"idx" : ["Books", "Science Fiction", "Three Body Problem", "Liu Cixin"]};

Requests Notes: Python HTTP API

A very pleasant HTTP library for Python.

"Requests takes all of the work out of Python HTTP/1.1 — making your integration with web services seamless. There’s no need to manually add query strings to your URLs, or to form-encode your POST data. Keep-alive and HTTP connection pooling are 100% automatic, powered by urllib3, which is embedded within Requests."

References

• Main Page, with good docs
• "SSL InsecurePlatform Error"

Incantations


blogodex = {"toc" : "requests", "idx" = ["python", "http"]};

Good post on testing server failures

tl;dr: make your system "pressure tolerant" by using system tools to simulate failures.

If there's one thing to know about distributed systems, it's that they have to be designed with the expectation of failure. It's also safe to say that most software these days is, in some form, distributed—whether it's a database, mobile app, or enterprise SaaS. If you have two different processes talking to each other, you have a distributed system, and it doesn't matter if those processes are local or intergalactically displaced. 

Marc Hedlund recently had a great post on Stripe's game-day exercises where they block off an afternoon, take a blunt instrument to their servers, and see what happens. We're talking like abruptly killing instances here—kill -9, ec2-terminate-instances, yanking on the damn power cord—that sort of thing. Everyone should be doing this type of stuff. You really don't know how your system behaves until you see it under failure conditions.

http://www.bravenewgeek.com/sometimes-kill-9-isnt-enough/