Mechanical Soup: A Python library for automating interaction with websites

MechanicalSoup automatically stores and sends cookies, follows redirects, and can follow links and submit forms. It doesn't do Javascript.

https://github.com/hickford/MechanicalSoup

Linux: Invalidating Cache

Here's the incantation for invalidating the linux file system cache.  Good for running performance tests when you don't want a hot cache throwing off the results.

sudo sh -c 'sync;echo 3 > /proc/sys/vm/drop_caches' 

Pymunk

"pymunk is a easy-to-use pythonic 2d physics library that can be used whenever you need 2d rigid body physics from Python."

https://github.com/viblo/pymunk

oracle snippet for sorting by ip address

select n from t1 order by regexp_replace(regexp_replace(n, '(\d+)', '00\1'), '0*(\d{3})', '\1');

Simple C Web Server

Here's a nice example of practical network programming in C, about 200 loc.

http://www.ibm.com/developerworks/systems/library/es-nweb/index.html

Two Interesting Articles on UDP

Should you use UDP in your game?  Here's two interesting points of view.

• http://1024monkeys.wordpress.com/2014/04/01/game-servers-udp-vs-tcp/
• https://thoughtstreams.io/glyph/your-game-doesnt-need-udp-yet/

James Iry’s history of programming languages (illustrated with pictures and large fonts)

Very funny!


http://thequickword.wordpress.com/2014/02/16/james-irys-history-of-programming-languages-illustrated-with-pictures-and-large-fonts/

Publishing via Github Pages

Here's two intros. Razius uses pelican, and 24ways uses jekyll, but the idea is the same.  Generate static HTML via some markup language, push to github, and it's viewable.  And of course, there's good instructions on the Github Pages site itself.

Update: maybe good for publishing via S3?

jq - like sed for JSON

Over on github, a potentially nifty JSON processor.

jq is like sed for JSON data – you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text. 

jq is written in portable C, and it has zero runtime dependencies. You can download a single binary, scp it to a far away machine, and expect it to work. 

jq can mangle the data format that you have into the one that you want with very little effort, and the program to do so is often shorter and simpler than you’d expect.

Nifty Pulp Magazine Cover Generator

http://thrilling-tales.webomator.com/derange-o-lab/pulp-o-mizer/pulp-o-mizer.html

Perfect for all your software projects!

Various Oracle Data Dictionary Queries

adapted from here.



TABLES

    select distinct lower(table_name) as table_name
      from user_tab_columns
  order by 1

SEQUENCES

    select lower(sequence_name) as sequence_name,
           min_value,
           max_value,
           increment_by,
           cycle_flag,
           order_flag,
           cache_size
      from user_sequences
  order by 1


FOREIGN KEYS

    select ucc.constraint_name,
           ucc.column_name,
           fc.table_name
      from user_cons_columns ucc,
           user_constraints fc,
           user_constraints uc
     where uc.constraint_type = 'R' and
           uc.constraint_name = ucc.constraint_name and
           fc.constraint_name = uc.r_constraint_name and
           uc.table_name='%s'
  order by 1, 2


DESC

    select column_name as name,
           data_type as type,
           char_length as length,
           nullable,
           data_default as "default"
      from user_tab_columns
     where table_name='%s'
  order by column_name


CONSTRAINTS

    select ucc.constraint_name,
           ucc.column_name,
           uc.constraint_type,
           uc.search_condition
      from user_constraints uc,
           user_cons_columns ucc
     where uc.constraint_name = ucc.constraint_name and
           uc.table_name='%s' and
           uc.constraint_type = 'C'
  order by ucc.constraint_name, ucc.position

TRIGGERS

    select trigger_name,
          trigger_type,
          triggering_event,
          table_name,
          description,
          trigger_body
     from user_triggers
 order by 1

LIST_INDEX

    select case
             when constraint_type = 'P' then 'PRIMARY KEY'
             else ' ' end as index_type,
           ui.index_name,
           ui.uniqueness,
           uic.column_name,
           uic.column_position,
           uic.descend
      from user_indexes ui
           join user_ind_columns uic on uic.index_name = ui.index_name
           left join user_constraints
             on user_constraints.constraint_name = ui.index_name and
             user_constraints.constraint_type = 'P'
     where ui.table_name = '%s'
  order by constraint_type, uic.column_position;

INDEX

    select case
             when constraint_type = 'P' then 'PRIMARY KEY'
             else ' ' end as index_type,
           ui.table_name,
           ui.index_name,
           ui.uniqueness,
           uic.column_name,
           uic.column_position,
           uic.descend
      from user_indexes ui
             join user_ind_columns uic on uic.index_name = ui.index_name
             left join user_constraints
               on user_constraints.constraint_name = ui.index_name and
               user_constraints.constraint_type = 'P'
     where ui.index_name = '%s'
  order by constraint_type, uic.column_position;

Interesting way to generate some Oracle sample tables

A lot of times you need to create a sample table with a primary and foreign key, populated with some kind of name-like field.  Here's an interesting way to do it.

create table emp_tab as
select
  rownum empno,
  object_name ename,
  mod(rownum, 10) + 1 deptno,
  rownum * 100 sal
from all_objects
where rownum < 100;



Now you can do the same thing, for the foreign key.

create table dept_tab as
select
  rownum deptno,
  'd_' || rownum  deptname
from all_objects
where rownum <= 10;




mh@templar> select * from emp_tab;
empno ename                       deptno sal  
----- -----                       ------ ---  
    1 DUAL                             2  100 
    2 DUAL                             3  200 
    3 SYSTEM_PRIVILEGE_MAP             4  300 
    4 SYSTEM_PRIVILEGE_MAP             5  400 
    5 TABLE_PRIVILEGE_MAP              6  500 
    6 TABLE_PRIVILEGE_MAP              7  600 

Latency numbers every programmer should know — Gist

Latency numbers every programmer should know — Gist:

L1 cache reference                          0.5 ns

Branch mispredict                             5 ns

L2 cache reference                            7 ns

Mutex lock/unlock                            25 ns

Main memory reference                       100 ns

Compress 1K bytes with Zippy              3,000 ns

Send 2K bytes over 1 Gbps network        20,000 ns

Read 1 MB sequentially from memory      250,000 ns

Round trip within same datacenter       500,000 ns

Disk seek                            10,000,000 ns

Read 1 MB sequentially from disk     20,000,000 ns

Send packet CA->Netherlands->CA     150,000,000 ns




By Jeff Dean (http://research.google.com/people/jeff/):

A note from Donald Knuth about The Art of Computer Programming

A while back I answered a question on Stack Overflow asking about the relative popularity of The Art of Computer Programming. I replied with something I had heard, that he had commented that they were the "most purchased, least read" computer book in the world.  One Stack Overflow user, ShreevatsaR, had the good sense to call BS on my unsourced attribution.



By way of contrition, I sent the attached note to Knuth (we had met a couple of times previously -- enough to give me some hope of a response) asking him to confirm or deny.  Attached is his response.  We followed up with lunch, and he spoke quite approvingly of Stack Overflow.  I think the Math Stack people at the time were working Tex in the Stack markup language, and they had been in communication with him as well.


 He responded right away:

I have to say that the phrase isn't particular apt, since there obviously exist computer science books less read than TAOCP.  It is true that the *ratio* of pages-thoroughly-read to pages-purchased is pretty low for TAOCP, because different people are interested in different pages.  But still I like to think that the pages read are so great that the buyers aren't unhappy overall. (Or else they just want their friends to think that they have mastered a lot of tough material, which I must confess is not easy for anybody I know including myself.  Einstein said it best: 'Make things as simple as possible but no simpler.'  Some theories are inherently unsimplifiable ... we can even *prove* that!)

My favorite related comment is the enclosed ad that ran about ten years ago, when it was rumored that John Grisham was thinking of responding by writing a seven-volume treatise.

Anyway the bottom line is that I'm enthused about sending a memo to The Internet about this particular phrase.  The web has thousands of tales that aren't true, about virtually everybody in public life.  (Including the story about me and Pixar's CEO.)  I *did* say things like 'Premature optimization is the root of all evil in programming' and 'Beware of bugs in the above code -- I've only proved it correct not actually run it.' 

I especially like that he marked up my original note and fixed a grammatical error!

Thanks again to the most excellent ShreevatsaR who prompted me to do this!

blog.bjrn.se: Let’s build an MP3-decoder!

blog.bjrn.se: Let’s build an MP3-decoder!: Even though MP3 is probably the single most well known file format and codec on Earth, it’s not very well understood by most programmers – for many encoders/decoders is in the class of software “other people” write, like standard libraries or operating system kernels. This article will attempt to demystify the decoder, with short top-down primers on signal processing and information theory when necessary. Additionally, a small but not full-featured decoder will be written (in Haskell), suited to play around with.

Oracle Packages that include Java

Here's a cheat sheet for including Java stored procedures  in packages.

Write a class:

    public class OracleFoo {
        public static void myproc(int port, String s) throws  Exception {
        }
    }

Compile and Load:

    javac OracleFoo.java
    loadjava  -u scott/tiger OracleFoo.class

Declare a package:

    create or replace package oraclefoo
    as
        procedure myproc(port number, s varchar2)
          as language java name 'OracleFoo.myproc(int, java.lang.String)';
    end;

Call:

    begin
        oracletbslow.myproc(7728,'[1,"a","b1",{}]');
        oracletbslow.myproc(7727,'[1,"a","b2",{}]');
    end;   

Permissions:

    need to set these, double-check notes...

Six Keys to Being Excellent at Anything - Tony Schwartz - The Conversation - Harvard Business Review

Six Keys to Being Excellent at Anything - Tony Schwartz - The Conversation - Harvard Business Review:

1. Pursue what you love.
2. Do the hardest work first.
3. Practice intensely
4. Seek expert feedback, in intermittent doses.
5. Take regular renewal breaks.
6. Ritualize practice.

An AskTom about database APIs

An interesting note by Tom regarding APIs in the database.

API's -- all about API's. The applications don't do table level stuff, the apps don't even really *know* about tables. I have a schema. On top of schema we have an API that does the data stuff. In the application we call this API but have our own application logic as well (only application logic is in the application, data logic is -- well, right where it belongs -- next to the data, waiting for the 'next great programming paradigm to come along') [...] Applications -- a dime a dozen. They come, they go, they have programming paradigm of the week. Data -- priceless. Most of it has been here for a long long long time. Most all of it will be here after we are dead.

What does "flatdown" mean?

It's referenced in the OPML spec. Here's an explanation:

First, flatdown, in UserLand's outliner terminology, is a direction. The complete list of directions is: up, down, left, right, flatup, flatdown and nodirection. They are used as parameters to the op.xxx verbs which take a direction as a parameter. The first four are structured directions. Going up from an outline element takes you to its previous sibling, down takes you to the next sibling. Left takes you to the parent, right takes you to the first child. The flat directions, flatup and flatdown view the outline as a linear sequence, not structured. Imagine the outline displayed on a screen. To go flatup you'd go to the outline element displayed immediately before, to go flatdown you'd go to the outline element displayed immediately after.

Note to a Software Vendor, regarding Document Managment Integration

Hi ----, Thanks for getting back so soon. I hope you can look at this issue over your entire product line. Here's the situation as I see it. Ancient Times:

• I had "my" computer, "my" files, I managed everything myself. It was all about me, Me, ME! If you wanted a file, I would mail it to you or copy it to a network share.

Modern Times:

• I have at least two computers, a desktop and a laptop. My files are under revision control and stored in a central repository. I sync files to my local box and work on them. Sometimes I work on the same file on two boxes -- my desktop coz it has a big screen, and my laptop coz I can take it to meetings. If you want a file, I can just tell you where it is on the central server and you can access it at your leisure. And good news for you, I have two copies each of [your product].

Most existing software works well in the first case. Software that works in the second case will, IMHO, start to replace software that does not, even if the other software is better in most other regards. As an example, we've installed [a web-based package] on our wiki. Butt ugly and limited, but certainly matching the modern time requirements and very "friction free" with regards to sharing files. Of course, it's hard for any local app to beat a web app in the sharing arena, but I hope that all your products become "so close" that there's no temptation for people to switch. As a practical thing, here's what I would love to see for all your products.

• easy to check in / check out in revision control system.

This means either

• - making documents into single files
• - adding support for various revision systems
• - adding hooks so revision system plugins can be created

• easy multi-machine usage.

This is a big deal for something like [the project tracker package]. I'm sad to say I've got two copies but I stopped using it, because I would leave it running on my desktop, and then I couldn't run it on my laptop at home. Or I would add some action items in a meeting on my laptop, close the lid, and at my desktop, not have them show up. For programs like this, you should look at how IMAP email works. It is very convenient, I have my desktop mail open, my laptop mail open, my PDA mail open, and no matter what I do they all just magically stay in sync. Anyways, all IMHO, of course, and I hope this is useful feedback to you. Regards, Mark