Writing plugins and scripts

[BarthaziAndras]

I would like to create a plugin/script (I don't really know what a script is for), that extracts data from a page.

Let's say:

var page = HTTPRequest("http://www.kirix.com/");

and let's get the meta information:

var results = [];
var metas = page.match(/<meta.*?>/g);
var i;
for(i=0; i<metas.length; i++) {
var name = metas[i].match(/name="(.*?)"/)[1];
var value = metas[i].match(/content="(.*?)"/)[1];
results.push({"name": name, "value": value});
}

This is JS (let's say it's working, I've just wrote it w/o testing). Is this code compatible with Kirix Strata? If I'm here, what should I do if I would like to insert the results array as a dataset into Strata? How can I add an "Import metas" entry into the context menu just after the "Import table"?

If I would like to create an "Import List" function, just to extract <ul><li> data, how I can access the actual site's DOM, and how I will know where the right clicking happened on the page.

Is it possible to write a plugin like this at all?

[Dave]

Hello,

So many good questions in your post...

We hope to provide you with a good example to get you going, however, in the meantime, here's some references to posts/documentation which hopefully will get you started:

Forum thread on plugins (or as we call them, "Extensions")
- viewtopic.php?f=20&t=529

Some documentation (more to come soon, hopefully) on database connections and table creation
- http://kirix.com/docs/DbConnection.html
- http://kirix.com/docs/DbResult.html

The execute() function allows you to execute raw SQL commands, so that should be enough to get you started.

Regarding the difference between scripts and extensions...

A script is just a single javascript file which can either be stored in your Kirix Strata project or externally on the hard drive somewhere. Scripts are more ad-hoc, meaning you can create a script to do something, however, you're going to have to run it yourself and if it references any other files or resources and those files are moved or deleted, your script is going to be broken. An extension is basically a nice way of wrapping up javascript files along with other resource files (bitmaps, text, etc.) that may be in the included scripts. Creating an extension also allows you to include metadata which can provide a title, description, version information, etc. for your extension. Extensions are convenient because they are portable (self-contained) and can be set to run when Kirix Strata starts up.

The forum post referenced above contains some further information about how to create an extension and also includes a basic extension example.

Hope this helps!

Dave.

[Ben]

I'd write that first line with a "new" as

Code: Select all

var page = new HTTPRequest("http://www.kirix.com/");

The inline regex's aren't working just yet (but they will be), so for now just use the RegExp object:

So instead of /[A-Za-z]+/ just use

Code: Select all

var pattern1 = new RegExp("[A-Za-z]+");

Other than that I think your script will run fine!

Best,
Ben

[BarthaziAndras]

OK, now I see that Project's datasets are database tables behind the scenes, which is a quite nice idea. My question is that what kind of database engine working in Strata? It looks like MySQL, but it's just a guess.

[Aaron]

We implemented a data engine that has both a native driver as well as "mount points" or connections that let you use other databases directly. Using this approach, you can "virtualize" data, allowing you to integrate and analyze data from databases, filesystems, and the web all in one place.

The native driver doesn't require any other data engine to function and has a number of useful qualities and features that include 1) a SQL API, 2) large data capacity; some people are currently using the engine with tables that have more than 100 million records, 3) speed; many data operations, such as sorting, filtering, grouping, perform quickly and scale well from small to large tables.

In addition, the data engine allows you to connect to other databases, such as Oracle or MySQL Enterprise, as well as files such as CSV, TSV, DBF and fixed-length files, and to manipulate and analyze the data contained in these with many of the same database operations supported natively.

For example, from the File menu, you can connect to Oracle or MySQL Enterprise, by clicking on the "Create Connection..." and entering the appropriate connection information for the database to which you are connecting. Once you've connected, the database appears in the project as a folder, and you can access these tables and perform many of the same operations on them as native tables. You can even mount multiple databases and file systems at the same time, allowing you to open tables from each database, and even export and import them from one to another by dragging and dropping the tables from one connection to another. So, if you want to import a CSV file into MySQL, connect to both the filesystem and to MySQL, then simply drag the CSV file from the filesystem folder into the MySQL folder.

In the scripting API, similar connections are possible: to connect to the native database, use DbConnection(); to connect to another database, use DbConnection() with an appropriate connection string.

[BarthaziAndras]

OK, I got it. Do you have any documentation about the SQL API of your database? I mean how to create a table? What are data types, INT, INT(14), INTEGER, or VARCHAR(255), CHAR(255), CHAR2(255), TEXT, BLOB should be written if I specify a column? SELECT in SELECT is possible (I guess not, not yet tried). What kind of function I can use, and what are they proper syntax? So, do you have any information about it's features you can provide me?

To be honest, I'm ok with my current knowledge now, but it would nice to give some answer to these questions.

[Ben]

Hello Andras,

The types are as follows:

VARCHAR(n) - 8-bit
CHAR(n) - 8-bit
NVARCHAR(n) - unicode character
NCHAR(n) - unicode character
INTEGER
NUMERIC(w,s) - numeric (width, scale/dec places)
DOUBLE
DATE
DATETIME
BOOLEAN

You can effectively do a SELECT in a SELECT with the following code:

db.execute
("
SELECT * INTO TABLE /.temp/tmptbl1 FROM source WHERE condition = 123;
SELECT * FROM /.temp/tmptbl1 WHERE condition = 456;
DROP TABLE IF EXISTS /.temp/tmptbl1;
");

Most 'standard' sql functions work. You can find a list of them with their syntaxes here: http://www.kirix.com/olddocs/Category:SQL_Functions. We're in the process of completing our documentation for the new version, and merging all of this in. We'll put a post out there when the whole thing is done.

All the best,
Ben