Oracle: quick AS OF demo

First, let's check our current SCN

    scott@orcl> select current_scn from v$database;
    current_scn
    -----------
    7559499070893

And see how many rows we have in table FOO

    scott@orcl> select count(*) from foo;
    count(*)
    --------
          33

Delete everything from FOO

    scott@orcl> delete from foo;
    (33 rows, 0.011 sec)

    scott@orcl> select count(*) from foo;
    count(*)
    --------
           0

What's our SCN now?

    scott@orcl> select current_scn from v$database;
    current_scn
    -----------
    7559499071397

What was the count as of yesterday?

    scott@orcl> select count(*) from foo as of timestamp sysdate-1;
    count(*)
    --------
          33

And as of the SCN at the start of our session?

    (1 rows, 0.010 sec)
    scott@orcl> select count(*) from foo as of scn 7559499070893;
    count(*)
    --------
          33

blogodex = {"toc" : "Oracle SCN", "idx" : ["Oracle", "SCN", "System Change Number", "as of"]};

Installing Maven

Pretty simple:

• Download tarball: https://maven.apache.org/download.cgi
• untar
• sudo mv apache-maven-3.3.0 /opt/maven
• add /opt/maven/bin to your path

This was on a Mac.

blogodex = {"idx" : ["maven", "installation"]};

SQL Developer Keepalive plugin

When you start SQL Developer, click on the icon.  Pressing enter gives you the default of sending a message every 600 seconds.

https://github.com/scristalli/SQL-Developer-4-keepalive

blogodex = {"idx" : {"SQL Developer", "Oracle connection timeout", "ORA-03135"]};

What if your name is actually Null?

Interesting comments by Mr. Christopher Null!

I'm adding "null" to the blogodex, will it come back to bite me?

http://www.wired.com/2015/11/null

blogodex = {"idx" : ["problem areas", "null"]};

Nice article on The Log

"Each working data pipeline is designed like a log; each broken data pipeline is broken in its own way."—Count Leo Tolstoy (translation by Jay Kreps)

Here's an excellent manifesto about "the data log".  You might be familiar with this concept in the guise of database journals, or with event sequences in distributed systems.

Jay starts from there and writes (pretty comprehensively!) about how this idea is used in several modern systems, and talks about his experience at linkedin using this approach.  Lots of interesting references at the end.  I was happy to see John Ousterhout mentioned there!

The Log: What every software engineer should know about real-time data's unifying abstraction

blogodex = {"idx" : ["logging", "Jay Kreps", "Kafka", "fundamentals", "scalability"]};

Some JVM Resources

Three interesting sources:

• JVM Bytecode for Dummies
• Create Your Own Programming Language
• Implement Your Own JVM Compiler

JVM Bytecode for Dummies

Nice presentation by Charles Nutter.

http://www.slideshare.net/CharlesNutter/redev-2010-jvm-bytecode-for-dummies

Monitoring Oracle Dataguard Status

Received a tip from the gurus:

ON STANDBY:
v$dataguard_stats - various stats, including apply lag.
v$dataguard_status - status messages related to dataguard.
v$managed_standby - info on the standby 

ON PRIMARY
v$archive_gap - shows the # of archived redo logs the standby is
behind.  Since we're shipping in real-time, this should nearly always be 0.

Boto 3 Released

Key points:

• works with python 2 and 3
• interoperable with boto
• new package name: boto3
• docs here: https://boto3.readthedocs.org/
• pip install boto3

https://aws.amazon.com/blogs/aws/now-available-aws-sdk-for-python-3-boto3/

Setting a Sphinx theme

To set a theme in Sphinx:

• find a theme you want to copy.  It will generally be a directory full of files.
• copy the directory under your project's _templates directory.
• set the html_theme in conf.py
• some docs mention setting html_theme_path as well, but it doesn't seem necessary if it's under _templates.
• There's some nice themes at https://readthedocs.org

Example:

$ ls _templates/

sphinx_rtd_theme/

$ diff conf.py-orig conf.py

113c113,114

< html_theme = 'alabaster'

---

> html_theme = "sphinx_rtd_theme"

> html_theme_path = ["_themes", ]    ### probably not necessary

"git push" to Github wants me to log in!

If git prompts you for a username when you push:

$ git push

Username: 

Check your remote settings.

$ git remote -v

origin https://github.com/YOURNAME/notebook.git (fetch)

origin https://github.com/YOURNAME/notebook.git (push)

If it's an https reference, you can change it to an ssh reference as follows. 

$ git remote set-url origin git@github.com:YOURNAME/notebook.git

And everything is great!

$ git push

Everything up-to-date

Mathematical Theory of Claude Shannon

Claude Shannon is well-known for having pioneered information theory and switching theory.  Here's a great overview of his approach to these and other areas he studied.

http://web.mit.edu/6.933/www/Fall2001/Shannon1.pdf

Software: Pagegen Static Site Generator

Pagegen converts a directory tree of RST docs into a static site.  "Aims to generate a fully featured site ready for CSS styling," according to the docs.

blogodex = {"toc" : "PageGen", "idx" = ["Software"]};

Book: The Three Body Problem, Liu Cixin

When I was in China, I met some Sci Fi enthusiasts and asked who their favorite authors were.  I wasn't sure who might have been translate, and how the various political points of view might have affected what was brought in to the country.

To my surprise, they had a lot of authors, all of them Chinese.  It turns out that the country's enthusiasm for science led to Science Fiction being very popular in the country.

I hadn't heard of any of them, and none of the works seem to have been translated into English. Perhaps this is going to change, starting with this book?

Amazon link and their blurb:

The Three-Body Problem is the first chance for English-speaking readers to experience this multiple award winning phenomenon from China's most beloved science fiction author, Liu Cixin. 

Set against the backdrop of China's Cultural Revolution, a secret military project sends signals into space to establish contact with aliens. An alien civilization on the brink of destruction captures the signal and plans to invade Earth. Meanwhile, on Earth, different camps start forming, planning to either welcome the superior beings and help them take over a world seen as corrupt, or to fight against the invasion. The result is a science fiction masterpiece of enormous scope and vision.

blogodex = {"idx" : ["Books", "Science Fiction", "Three Body Problem", "Liu Cixin"]};

Requests Notes: Python HTTP API

A very pleasant HTTP library for Python.

"Requests takes all of the work out of Python HTTP/1.1 — making your integration with web services seamless. There’s no need to manually add query strings to your URLs, or to form-encode your POST data. Keep-alive and HTTP connection pooling are 100% automatic, powered by urllib3, which is embedded within Requests."

References

• Main Page, with good docs
• "SSL InsecurePlatform Error"

Incantations


blogodex = {"toc" : "requests", "idx" = ["python", "http"]};

Good post on testing server failures

tl;dr: make your system "pressure tolerant" by using system tools to simulate failures.

If there's one thing to know about distributed systems, it's that they have to be designed with the expectation of failure. It's also safe to say that most software these days is, in some form, distributed—whether it's a database, mobile app, or enterprise SaaS. If you have two different processes talking to each other, you have a distributed system, and it doesn't matter if those processes are local or intergalactically displaced. 

Marc Hedlund recently had a great post on Stripe's game-day exercises where they block off an afternoon, take a blunt instrument to their servers, and see what happens. We're talking like abruptly killing instances here—kill -9, ec2-terminate-instances, yanking on the damn power cord—that sort of thing. Everyone should be doing this type of stuff. You really don't know how your system behaves until you see it under failure conditions.

http://www.bravenewgeek.com/sometimes-kill-9-isnt-enough/

Two Tcl Oldies...

I was surprised to see these bits of fiddlefaddle online:

The Tao of Tcl
A Class Code Generator

Building RESTful APIs with Tornado

Nice Tornado Article.

http://www.drdobbs.com/open-source/building-restful-apis-with-tornado/240160382

from datetime import date

import tornado.escape

import tornado.ioloop

import tornado.web

 

class VersionHandler(tornado.web.RequestHandler):

    def get(self):

        response = { 'version': '3.5.1',

                     'last_build':  date.today().isoformat() }

        self.write(response)

 

class GetGameByIdHandler(tornado.web.RequestHandler):

    def get(self, id):

        response = { 'id': int(id),

                     'name': 'Crazy Game',

                     'release_date': date.today().isoformat() }

        self.write(response)

 

application = tornado.web.Application([

    (r"/getgamebyid/([0-9]+)", GetGameByIdHandler),

    (r"/version", VersionHandler)

])

 

if __name__ == "__main__":

    application.listen(8888)

    tornado.ioloop.IOLoop.instance().start()

The Night Watch

One of the most awesome computer papers ever written!

http://research.microsoft.com/en-us/people/mickens/thenightwatch.pdf

When you debug a distributed system or an OS kernel, you do it Texas-style. You gather some mean, stoic people, people who have seen things die, and you get some primitive tools, like a compass and a rucksack and a stick that’s pointed on one end, and you walk into the wilderness and you look for trouble, possibly while using chewing tobacco. As a systems hacker, you must be prepared to do savage things, unspeakable things, to kill runaway threads with your bare hands, to write directly to network ports using telnet and an old copy of an RFC that you found in the Vatican. When you debug systems code, there are no high-level debates about font choices and the best kind of turquoise, because this is the Old Testament, an angry and monochromatic world, and it doesn’t matter whether your Arial is Bold or Condensed when people are covered in boils and pestilence and Egyptian pharaoh oppression. HCI people discover bugs by receiving a concerned email from their therapist. Systems people discover bugs by waking up and discovering that their first-born children are missing and “ETIMEDOUT” has been written in blood on the wall.

Mechanical Soup: A Python library for automating interaction with websites

MechanicalSoup automatically stores and sends cookies, follows redirects, and can follow links and submit forms. It doesn't do Javascript.

https://github.com/hickford/MechanicalSoup

Linux: Invalidating Cache

Here's the incantation for invalidating the linux file system cache.  Good for running performance tests when you don't want a hot cache throwing off the results.

sudo sh -c 'sync;echo 3 > /proc/sys/vm/drop_caches' 

Pymunk

"pymunk is a easy-to-use pythonic 2d physics library that can be used whenever you need 2d rigid body physics from Python."

https://github.com/viblo/pymunk

oracle snippet for sorting by ip address

select n from t1 order by regexp_replace(regexp_replace(n, '(\d+)', '00\1'), '0*(\d{3})', '\1');

Simple C Web Server

Here's a nice example of practical network programming in C, about 200 loc.

http://www.ibm.com/developerworks/systems/library/es-nweb/index.html

Two Interesting Articles on UDP

Should you use UDP in your game?  Here's two interesting points of view.

• http://1024monkeys.wordpress.com/2014/04/01/game-servers-udp-vs-tcp/
• https://thoughtstreams.io/glyph/your-game-doesnt-need-udp-yet/

James Iry’s history of programming languages (illustrated with pictures and large fonts)

Very funny!


http://thequickword.wordpress.com/2014/02/16/james-irys-history-of-programming-languages-illustrated-with-pictures-and-large-fonts/

Publishing via Github Pages

Here's two intros. Razius uses pelican, and 24ways uses jekyll, but the idea is the same.  Generate static HTML via some markup language, push to github, and it's viewable.  And of course, there's good instructions on the Github Pages site itself.

Update: maybe good for publishing via S3?

jq - like sed for JSON

Over on github, a potentially nifty JSON processor.

jq is like sed for JSON data – you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text. 

jq is written in portable C, and it has zero runtime dependencies. You can download a single binary, scp it to a far away machine, and expect it to work. 

jq can mangle the data format that you have into the one that you want with very little effort, and the program to do so is often shorter and simpler than you’d expect.

Nifty Pulp Magazine Cover Generator

http://thrilling-tales.webomator.com/derange-o-lab/pulp-o-mizer/pulp-o-mizer.html

Perfect for all your software projects!

Various Oracle Data Dictionary Queries

adapted from here.



TABLES

    select distinct lower(table_name) as table_name
      from user_tab_columns
  order by 1

SEQUENCES

    select lower(sequence_name) as sequence_name,
           min_value,
           max_value,
           increment_by,
           cycle_flag,
           order_flag,
           cache_size
      from user_sequences
  order by 1


FOREIGN KEYS

    select ucc.constraint_name,
           ucc.column_name,
           fc.table_name
      from user_cons_columns ucc,
           user_constraints fc,
           user_constraints uc
     where uc.constraint_type = 'R' and
           uc.constraint_name = ucc.constraint_name and
           fc.constraint_name = uc.r_constraint_name and
           uc.table_name='%s'
  order by 1, 2


DESC

    select column_name as name,
           data_type as type,
           char_length as length,
           nullable,
           data_default as "default"
      from user_tab_columns
     where table_name='%s'
  order by column_name


CONSTRAINTS

    select ucc.constraint_name,
           ucc.column_name,
           uc.constraint_type,
           uc.search_condition
      from user_constraints uc,
           user_cons_columns ucc
     where uc.constraint_name = ucc.constraint_name and
           uc.table_name='%s' and
           uc.constraint_type = 'C'
  order by ucc.constraint_name, ucc.position

TRIGGERS

    select trigger_name,
          trigger_type,
          triggering_event,
          table_name,
          description,
          trigger_body
     from user_triggers
 order by 1

LIST_INDEX

    select case
             when constraint_type = 'P' then 'PRIMARY KEY'
             else ' ' end as index_type,
           ui.index_name,
           ui.uniqueness,
           uic.column_name,
           uic.column_position,
           uic.descend
      from user_indexes ui
           join user_ind_columns uic on uic.index_name = ui.index_name
           left join user_constraints
             on user_constraints.constraint_name = ui.index_name and
             user_constraints.constraint_type = 'P'
     where ui.table_name = '%s'
  order by constraint_type, uic.column_position;

INDEX

    select case
             when constraint_type = 'P' then 'PRIMARY KEY'
             else ' ' end as index_type,
           ui.table_name,
           ui.index_name,
           ui.uniqueness,
           uic.column_name,
           uic.column_position,
           uic.descend
      from user_indexes ui
             join user_ind_columns uic on uic.index_name = ui.index_name
             left join user_constraints
               on user_constraints.constraint_name = ui.index_name and
               user_constraints.constraint_type = 'P'
     where ui.index_name = '%s'
  order by constraint_type, uic.column_position;

Interesting way to generate some Oracle sample tables

A lot of times you need to create a sample table with a primary and foreign key, populated with some kind of name-like field.  Here's an interesting way to do it.

create table emp_tab as
select
  rownum empno,
  object_name ename,
  mod(rownum, 10) + 1 deptno,
  rownum * 100 sal
from all_objects
where rownum < 100;



Now you can do the same thing, for the foreign key.

create table dept_tab as
select
  rownum deptno,
  'd_' || rownum  deptname
from all_objects
where rownum <= 10;




mh@templar> select * from emp_tab;
empno ename                       deptno sal  
----- -----                       ------ ---  
    1 DUAL                             2  100 
    2 DUAL                             3  200 
    3 SYSTEM_PRIVILEGE_MAP             4  300 
    4 SYSTEM_PRIVILEGE_MAP             5  400 
    5 TABLE_PRIVILEGE_MAP              6  500 
    6 TABLE_PRIVILEGE_MAP              7  600 

Latency numbers every programmer should know — Gist

Latency numbers every programmer should know — Gist:

L1 cache reference                          0.5 ns

Branch mispredict                             5 ns

L2 cache reference                            7 ns

Mutex lock/unlock                            25 ns

Main memory reference                       100 ns

Compress 1K bytes with Zippy              3,000 ns

Send 2K bytes over 1 Gbps network        20,000 ns

Read 1 MB sequentially from memory      250,000 ns

Round trip within same datacenter       500,000 ns

Disk seek                            10,000,000 ns

Read 1 MB sequentially from disk     20,000,000 ns

Send packet CA->Netherlands->CA     150,000,000 ns




By Jeff Dean (http://research.google.com/people/jeff/):

A note from Donald Knuth about The Art of Computer Programming

A while back I answered a question on Stack Overflow asking about the relative popularity of The Art of Computer Programming. I replied with something I had heard, that he had commented that they were the "most purchased, least read" computer book in the world.  One Stack Overflow user, ShreevatsaR, had the good sense to call BS on my unsourced attribution.



By way of contrition, I sent the attached note to Knuth (we had met a couple of times previously -- enough to give me some hope of a response) asking him to confirm or deny.  Attached is his response.  We followed up with lunch, and he spoke quite approvingly of Stack Overflow.  I think the Math Stack people at the time were working Tex in the Stack markup language, and they had been in communication with him as well.


 He responded right away:

I have to say that the phrase isn't particular apt, since there obviously exist computer science books less read than TAOCP.  It is true that the *ratio* of pages-thoroughly-read to pages-purchased is pretty low for TAOCP, because different people are interested in different pages.  But still I like to think that the pages read are so great that the buyers aren't unhappy overall. (Or else they just want their friends to think that they have mastered a lot of tough material, which I must confess is not easy for anybody I know including myself.  Einstein said it best: 'Make things as simple as possible but no simpler.'  Some theories are inherently unsimplifiable ... we can even *prove* that!)

My favorite related comment is the enclosed ad that ran about ten years ago, when it was rumored that John Grisham was thinking of responding by writing a seven-volume treatise.

Anyway the bottom line is that I'm enthused about sending a memo to The Internet about this particular phrase.  The web has thousands of tales that aren't true, about virtually everybody in public life.  (Including the story about me and Pixar's CEO.)  I *did* say things like 'Premature optimization is the root of all evil in programming' and 'Beware of bugs in the above code -- I've only proved it correct not actually run it.' 

I especially like that he marked up my original note and fixed a grammatical error!

Thanks again to the most excellent ShreevatsaR who prompted me to do this!

blog.bjrn.se: Let’s build an MP3-decoder!

blog.bjrn.se: Let’s build an MP3-decoder!: Even though MP3 is probably the single most well known file format and codec on Earth, it’s not very well understood by most programmers – for many encoders/decoders is in the class of software “other people” write, like standard libraries or operating system kernels. This article will attempt to demystify the decoder, with short top-down primers on signal processing and information theory when necessary. Additionally, a small but not full-featured decoder will be written (in Haskell), suited to play around with.

Oracle Packages that include Java

Here's a cheat sheet for including Java stored procedures  in packages.

Write a class:

    public class OracleFoo {
        public static void myproc(int port, String s) throws  Exception {
        }
    }

Compile and Load:

    javac OracleFoo.java
    loadjava  -u scott/tiger OracleFoo.class

Declare a package:

    create or replace package oraclefoo
    as
        procedure myproc(port number, s varchar2)
          as language java name 'OracleFoo.myproc(int, java.lang.String)';
    end;

Call:

    begin
        oracletbslow.myproc(7728,'[1,"a","b1",{}]');
        oracletbslow.myproc(7727,'[1,"a","b2",{}]');
    end;   

Permissions:

    need to set these, double-check notes...

Six Keys to Being Excellent at Anything - Tony Schwartz - The Conversation - Harvard Business Review

Six Keys to Being Excellent at Anything - Tony Schwartz - The Conversation - Harvard Business Review:

1. Pursue what you love.
2. Do the hardest work first.
3. Practice intensely
4. Seek expert feedback, in intermittent doses.
5. Take regular renewal breaks.
6. Ritualize practice.

An AskTom about database APIs

An interesting note by Tom regarding APIs in the database.

API's -- all about API's. The applications don't do table level stuff, the apps don't even really *know* about tables. I have a schema. On top of schema we have an API that does the data stuff. In the application we call this API but have our own application logic as well (only application logic is in the application, data logic is -- well, right where it belongs -- next to the data, waiting for the 'next great programming paradigm to come along') [...] Applications -- a dime a dozen. They come, they go, they have programming paradigm of the week. Data -- priceless. Most of it has been here for a long long long time. Most all of it will be here after we are dead.

What does "flatdown" mean?

It's referenced in the OPML spec. Here's an explanation:

First, flatdown, in UserLand's outliner terminology, is a direction. The complete list of directions is: up, down, left, right, flatup, flatdown and nodirection. They are used as parameters to the op.xxx verbs which take a direction as a parameter. The first four are structured directions. Going up from an outline element takes you to its previous sibling, down takes you to the next sibling. Left takes you to the parent, right takes you to the first child. The flat directions, flatup and flatdown view the outline as a linear sequence, not structured. Imagine the outline displayed on a screen. To go flatup you'd go to the outline element displayed immediately before, to go flatdown you'd go to the outline element displayed immediately after.

Note to a Software Vendor, regarding Document Managment Integration

Hi ----, Thanks for getting back so soon. I hope you can look at this issue over your entire product line. Here's the situation as I see it. Ancient Times:

• I had "my" computer, "my" files, I managed everything myself. It was all about me, Me, ME! If you wanted a file, I would mail it to you or copy it to a network share.

Modern Times:

• I have at least two computers, a desktop and a laptop. My files are under revision control and stored in a central repository. I sync files to my local box and work on them. Sometimes I work on the same file on two boxes -- my desktop coz it has a big screen, and my laptop coz I can take it to meetings. If you want a file, I can just tell you where it is on the central server and you can access it at your leisure. And good news for you, I have two copies each of [your product].

Most existing software works well in the first case. Software that works in the second case will, IMHO, start to replace software that does not, even if the other software is better in most other regards. As an example, we've installed [a web-based package] on our wiki. Butt ugly and limited, but certainly matching the modern time requirements and very "friction free" with regards to sharing files. Of course, it's hard for any local app to beat a web app in the sharing arena, but I hope that all your products become "so close" that there's no temptation for people to switch. As a practical thing, here's what I would love to see for all your products.

• easy to check in / check out in revision control system.

This means either

• - making documents into single files
• - adding support for various revision systems
• - adding hooks so revision system plugins can be created

• easy multi-machine usage.

This is a big deal for something like [the project tracker package]. I'm sad to say I've got two copies but I stopped using it, because I would leave it running on my desktop, and then I couldn't run it on my laptop at home. Or I would add some action items in a meeting on my laptop, close the lid, and at my desktop, not have them show up. For programs like this, you should look at how IMAP email works. It is very convenient, I have my desktop mail open, my laptop mail open, my PDA mail open, and no matter what I do they all just magically stay in sync. Anyways, all IMHO, of course, and I hope this is useful feedback to you. Regards, Mark

begin
dbms_scheduler.create_job(
job_name => 'DEMO_JOB_SCHEDULE'
,job_type => 'PLSQL_BLOCK'
,job_action => 'begin insert into jobtest(ztime) values(systimestamp); end; '
,start_date => trunc(sysdate, 'CC')
,repeat_interval => 'FREQ=MINUTELY'
,enabled => TRUE
,comments => 'Demo for job schedule.');
end;

begin
DBMS_SCHEDULER.DROP_JOB('DEMO_JOB_SCHEDULE');
end;

-- show all jobs

select *
from dba_scheduler_jobs
where owner='MHTEST';


-- job history

select log_date
, job_name
, status
from dba_scheduler_job_log
where owner='MHTEST';

-- running jobs

select job_name
, session_id
, running_instance
, elapsed_time
, cpu_used
from dba_scheduler_running_jobs

using dbms_application_info

exec DBMS_APPLICATION_INFO.SET_MODULE( NULL,NULL );
exec DBMS_APPLICATION_INFO.SET_MODULE('my module', 'my action');
exec DBMS_APPLICATION_INFO.SET_ACTION('my action 2');
exec DBMS_APPLICATION_INFO.SET_CLIENT_INFO('my client info');

-- exec DBMS_APPLICATION_INFO.SET_SESSION_LONGOPS(rindex, slno,
--      'Test Operation', obj, 0, sofar, totalwork, 'Contract', 'Contracts');

select sid,program,module,action,client_info from v$session where username='MH'; 

Oracle/PLSQL: Oracle System Tables

Oracle/PLSQL: Oracle System Tables: "an alphabetical listing of the Oracle system tables that are commonly used."

Desc - Describe an Oracle Table, View, Synonym, package or Function

Desc - Describe an Oracle Table, View, Synonym, package or Function: "An alternative to the DESC command is selecting directly from the data dictionary - DESC MY_TABLE is equivalent to SELECT column_name 'Name', nullable 'Null?', concat(concat(concat(data_type,'('),data_length),')') 'Type' FROM user_tab_columns WHERE table_name='TABLE_NAME_TO_DESCRIBE';"