1. Anno-J Browser and jBuilder tool

A. System requirements

1) Linux/MacOS system.

2) httpd web service (Apache2) enable

3) PHP enable

4) MySQL server enable (optional)

B. Download & Installation

1) Downloads

a) Download jBuilder tool : http://signal.salk.edu/aj2/

Which contains jBuilder tool as well as the latest Anno-J browser scripts.

b) Download SimpleDB tool : http://signal.salk.edu/aj2/

Whcih contains SimpleDB tool and jBuilder Lite tool.

2) jBuilder Installation

[Linux]

> tar xfz aj2.tar.gz
> mkdir /srv/www/htdocs/aj2
> rsync -auv aj2/ /srv/www/htdocs/aj2
> chown -R wwwrun /srv/www/htdocs/aj2

[MacOS]

> mkdir /Library/Webserver/Documents/aj2
> rsync -auv aj2/ /Library/Webserver/Documents/aj2
> chown -R _www /Library/Webserver/Documents/aj2

3) SimpleDB Tool Installation

> tar xfz simple_db_bin.tar.gz

If there is a permission problem, please use sudo or login as su (super user).

4) Initialization

Before initializing aj2 builder, you need to create cache directory, proxy cache directory and change their owner or mode. These directories should have huge spaces available if you plan to use MySQL database. However, the cache directory is not important for SimpleDB.

> mkdir /home/hchen/public_html/aj2_caches
> mkdir /home/hchen/public_html/aj2_proxy_caches
> chown wwwrun /home/hchen/public_html/aj2_*caches
> chmod 755 /home/hchen/public_html/aj2_*caches [if need]

Then open the web browser : http://server/aj2/tools/init.php, and fill information need in the initialization table:

Cache Dir : /home/hchen/public_html/aj2_caches
Proxy Cache Dir : /home/hchen/public_html/aj2_proxy_caches
MySQL Server : primary server, like localhost
MySQL User : mysql_name
MySQL Password : mysql_passcode. If you have multiple servers, please set them to allow the same user to visit.
Admin ID : your_login_id
Admin Name : your_real_name
Email : your_email
Group : admin or admin,pbio
Priority: 7770. Please not set to 0777 or any ones start with 0, otherwise, you will not allow to login.

Now your jBuilder system is set up. You can open http://server/aj2/tools/index.php to login. Your first input will be remembered.

C. Administration

As a member of admin group, you are responsible to add user, build genome fetcher, build gene model fetcher, register a group, register an organism, add a mysql server, register genome url and register gene model url. You can add a user to admin group and thus who could share your duties.

1) Add a new user :

Priority -> 4 digitals of 7 or 0. 7??? a normal user. 0??? visitor. ?7?? group shares, ??7? all shares or ???7 cross origin resource shares (CORS).

Group : user can be assigned to multiple groups : admin,pbio,gale. A user can login to a specific group as user:gale, user:pbio.

admin group - administrator, with administration responsibility and capability.
super group - supervisor, with ability to open all user's Anno-J pages but without admin's priority.

2) Create user groups, register organisms, register existed genomes, existed gene models and mySQL servers for jBuilder users' access convenience.

3) Create genome fetchers.

-> Build Genomes:

1]. Collect genome sequences, and divide them to individual files based on chromosome or assembly. Each file contains nucleotide sequences only as well as from the first base to the last one. There is no title, no break line, no other special or hidden characters included. For example, the arabidopsis genomes can be split into files as chr1, chr2, chr3, chr4, chr5, chrM and chrC within the folder /path/Col-0/.

2]. Gather genome source information. You are encouraged to give credits to whom sequenced, assembled, and curated the genome.

3]. Login as admin [hchen:admin], first go to 'jAdmins' -> 'Organism' to register your species if not list, and then click on 'Build Genomes' to fill the 'Build Genome' table.

4]. If you want to display the chromosomes as 1, 2, 3, 4, 5, M and C in Anno-J browser 'Locations', the input for 'Genome Path:' would be '/path/Col-0/chr'.

5]. Save your genome file as: like ath_col-0_v10.php. It will generate a genome file within the folder your server's 'Documents_Root'./aj2/genomes. This genome file contains information you provide and each chromosome's name and length (size).

6]. Register your newly built genome in the 'jAdmins' -> 'Genome URLs'.

4) Create gene model fetchers.

-> Build Models

1]. Get genome annotation file (usually as gff3 file) and related information.

2]. Use simpleDB tool to build a SimpleModels track directly from gff3.

3]. If step 2 failed, convert gff3 to SimpleText, revise it and then use simpleDB tool to build a SimpleModels track from SimpleText.

4]. Login as admin [hchen:admin], click on 'Build Models' and fill the table. Please provide information about authors who generated the gff3 file and give them credits, fill server and table if models track is from MySQL database, or fill 'Table' with your SimpleModels track path and leave MySQL 'Server' blank if SimpleDB folder used.

5]. Register the gene models you build thru ->jAdmins -> Gene Models. Name for the gene models track would be 'Gene Models ($Alias)'.

5) Create proxy page, cors page source.

D. Build Your Page

1) Before building your own page

a) Prepare your data.

b) Convert your data to SimpleText format (see section 2: SimpleDB tools)

c) Formatting your data into simpleDB (see section 2: SimpleDB tools)

d) Create a suffix settings. You can use many times once you create it.

After your login, create a suffix (part of parameters or other information)

-> Settings -> Suffix Information (fill the form) ( -> means click)

Name : Your name
e-mail :
Institution : [Salk in default]
Institution Link : [Salk link in default]
Institution Logo : [Salk logo in default]
Organism - Genome URL : select one
Suffix Name : input like [Arabidopsis thaliana (TAIR 10)] or something

-> Submit

e) Create a default Simple Home Directory or Directories (SHD)

-> Settings -> Simple Home Directory
Directory : /home/simpleDB/hchen/arab [I have put the formatted or cache data in that directory]
-> Add Path

f) Google Analytics Script if you have one or leave it blank.

2) Build your page by using jBuilder tool

-> jBuilder [There are two panels and a column of icons in the far right]

These icons are respectively Edit, Sort, Active, Add (node), Delete (node), Expand/Collapse, Save - Rollback, Trash and Transfer button.
The left panel is the source panel and the right is the working panel.

a) Create Kay_CCA1 page and add Peaks, Beds, and Reads folders

-> Add (Click Add button)

+ Tracks

- NewPage [ -> NewPage to select it and allow it to be text editable ]
- Kay_CCA1 [ text NewPage changed to Kay_CCA1 - which is your page URL ]
[Click Add three times to add three New folders and Annotation Models folder will be added automatically]

- Annotation Models
- New Folder [-> click to change it to Peaks]
- New Folder [-> click to change it to Beds]
- New Folder [-> click to change it to Reads]

b) Add Annotation Models

- Select Annotation Models folder on the right panel by clicking it.

-> Source -> tables -> Gene Models -> Arabidopsis Models TAIR 10 [on the left panel]

Drag and drop "Gene Models (TAIR 10)" to Annotation Models folder (on the right panel) or rightArrow to transfer.

c) Add Peaks

- Select the Peaks folder

-> Source -> Tables -> Simple -> /home/simpleDB/hchen/arab -> Colleen

- Select "LD peaks" and use rightArrow to transfer it to Peaks folder. After LD transferred, just click rightArrow again and it will transfer LL the next one.

d) Add Beds and Reads [Repeat c to add bed and read tracks, ignore those you do not want to add]

e) Add AnnotatedPeaks to Annotation Models folders [which are searchable]

f) Add Suffix to Kay_CCA1 page

- Select the Kay_CCA1

-> Source -> Tables -> Suffixes -> Suffix file you just created in step 2. [rightArrow or Drag drop to transfer]

g) Expand and Save

h) Edit it if there is a problem and then repeat step g).

3) Build your page by using jBuilder Lite tool.

You can build a page by downloading simple_db_bin.tar.gz and using its Lite tools only. However, you might need the jBuilder tool to build genome fetcher or others.

a) Prepare your data.
b) Format your data into simpleDB database with right track type.
c) Copy pages/test.suffix to yours.suffix and edit it.
d) Follow pages/test.table to build your own table : yours.table.
e) Copy pages/test.php to yours.php and move to web directory. Edit it.
f) Test it : http://yourserver.com/yours.php?action=test .
g) Lauch it : http://yourserver.com/yours.php .

2. SimpleDB database and SimpleDB tool

SimpleDB tool, which is included in simple_db_bin, is a PHP program to convert text (SimpleText), bam, sam or bed file to json encoding and gzipped simpleDB database. The SimpleDB files can be directly loaded and used by Anno-J browser.

A. Data types:

1) SimpleModels, SimpleMask, SimpleAligns : tab delimited text table file with 6-8 columns

a) id : unique [important]
b) assembly : chromosome number [important]
c) strand : +/- [important]
d) start : start position [important]
e) end : end position [important]
f) class : [important] mRNA, five_prime_UTR, three_prime_UTR, Exon, red, green, purple, orange, MULE ...
g) parent_id : parent_id [!important]
h) description : null, description or annotations [!important]

SimpleDB syntax:

php simple_db.php -t SimpleModels -i src.txt -o dest.folder -p "100:1,10:1"
php simple_db.php -t SimpleMask -i src.txt -o dest.folder -p "100:1,10:1"
php simple_db.php -t SimpleAligns -i src.txt -o dest.folder -p "100:1,10:1"

2) SimpleReads, SimpleSmallReads : tab delimited text table file with 6-7 columns

a) id : unique [important for box and sequence display]
b) assembly : chromosome number [important]
c) strand : +/- [important]
d) start : start position [important]
e) end : end position [important]
f) sequence : read sequences [important]
g) copies : number of the sequences [!important, default: 1]

SimpleDB syntax:

php simple_db.php -t SimpleReads -i src.txt -o dest.folder
php simple_db.php -t SimpleSmallReads -i src.txt -o dest.folder

3) SimplePairedEnd : tab delimited text table file with 7-8 columns

a) id : unique [important for box and sequence display]
b) assembly : chromosome number [important]
c) strand : +/- [important]
d) start : start position [important]
e) end : end position [important]
f) seqA : read sequences [important]
g) seqB : sequence B [important].
h) copies : number of the sequences [!important, default: 1]

SimpleDB syntax:

php simple_db.php -t SimplePairedEnd -i src.txt -o dest.folder

4) SimpleIntensity : tab delimited text table with 6-7 columns

a) id : unique [!important]
b) assembly : chromosome number [important]
c) strand : +/- [important]
d) start : start position [important]
e) end : end position [important]
f) class : data type [!important, default: null = read]
g) copies : number of the sequences [!important, default: 1]

SimpleDB syntax:

php simple_db.php -t SimpleIntensity -i src.txt -o dest.folder

5) SimpleMeth : tab delimited text table with 8 columns

a) id : unique [important for box and sequence display]
b) assembly : chromosome [important]
c) strand : +/- [important]
d) start : start position [important]
e) end : end position [important]
f) class : CG, CHG or CHH
g) mC : number of mCs
h) h : total number of mCs + Cs

or tab delimited mC text table with 6 columns

a) assembly : chromosome [important]
b) position : position [important]
c) strand : +/- [important]
d) class : CG, CHG or CHH [important]
e) mC : number of mCs [important]
f) h : total number of mCs + Cs [important]

SimpleDB syntax:

php simple_db.php -t SimpleMeth -i src.txt -o dest.folder
php simple_db.php -t SimpleMeth -i mC.tsv -o dest.folder -d mC

6) SimpleSNPs : tab delimited text table with 7 columns

SimpleDB syntax:

php simple_db.php -t SimpleSNPs -i src.txt -o dest.folder

7) Bam, Sam files : output from samtools or bowtie alignment tool.

Bam, sam file can be converted to SimpleReads, SimpleSmallReads and SimplePairedEnd type.

SimpleDB syntax:

php simple_db.php -t SimpleReads -i src.bam -o dest.folder -d bam
php simple_db.php -t SimpleSmallReads -i src.bam -o dest.folder -d bam
php simple_db.php -t SimplePairedEnd -i src.sam -o dest.folder -d sam

8) Bed file : tab delimited txt table file with 5 or more columns

1) assembly : chromosome [important] - The name of the chromosome
2) start : position [important] - The starting position of the feature in the chromosome
3) end : position [important] - The ending position of the feature in the chromosome
4) name : name or . [important] - Defines the name of the BED line.
5) value : value or . [!important] - A score between 0 and 1000 or any.
6) strand : +/- or . [!important] - Defines the strand. Either "+" or "-" or ".".
7) thickStart : [!important] - The starting position at which the feature is drawn thickly.
8) thickEnd : [!important] - The ending position at which the feature is drawn thickly.
9) itemRgb : [!important] - An RGB value of the form R,G,B (e.g. 255,0,0).
10) blockCount : [!important] - The number of blocks (exons) in the BED line.
11) blockSizes : [!important] - A comma-separated list of the block sizes.
12) blockStarts : [!important] - A comma-separated list of block starts.
13) description : [!important] gene annotation or data description.
14) description : [!important] more description

Bed file can be converted into data type : SimpleModels, SimpleMask or SimpleIntensity.

Wig file : convert your wig to bed file or SimpleText file first.

SimpleDB syntax:

php simple_db.php -t SimpleIntensity -i src.bed -o dest.folder -d bed

9) Bionano q.cmap, r.cmap and x.xmap files : bionano output file types

SimpleNano syntax: Covert bionano files to SimpleText files

php simple_nano.php -r r.cmap -q q.cmap -x x.xmap -o out [-p 1,2-3,4,5]
The output SimpleText files: out.reference, out.query, out.aligns, out.coverage and out.query_normalized.

SimpleDB syntax:

php simple_db.php -t SimpleAligns -i out.aligns -o aligns
php simple_db.php -t SimpleMask -i out.reference -o refers
php simple_db.php -t SimpleModels -i out.query[_normalized] -o query[_normalized] -k keepSmall
php simple_db.php -t SimpleIntensity -i out.coverage -o coverage

10) Others : models.txt, walks.txt

models.txt: tab delimited and 5 column table, generated from simple_db.php -t SimpleModels and searchable by lookup and describe actions of ModelsTrack.

walks.txt: tab delimited and 4 column table, searchable by walk action if showWalks is true.

a) id : unique [important]
b) assembly : chromosome number [important]
c) start : start position [important]
d) end : end position [important]
e) description : text description

B. Usage:

php simple_db.php -t Data_Type -i Source_file -o Destination_directory [ ... [ ... ]]

php simple_db.php --type data_type --input src --output dest --policy resn --coversion cvsn [ ... [ ... ]]

-t,--type : data_type, one of SimpleModels, SimpleMask, SimpleReads, SimplePairedEnd, SimpleMeth, SimpleSmallReads, SimpleSNPs, SimpleAligns, SimpleIntensity or SimpleMicroarray

-i,--input : value : src/src_file = input file. .bam .sam .bed or Simple text file.

-o,--output : value : dest/dest_dir = output cache directory. It will be used by jBuilder.

-p,--policy : value : resn = full or sub set of "1000:1, 100:1, 10:1, 1:1, 1:10, 1:100". To change cache size, set "1:100:10, 10:1:100" to increase 10 or 100 times.

-c,--coversion : value : convert chr names to anno-j names. cvsn = "chr1:1, mitochrondia:M, Chr:".

-d,--dataType : input file type [accept bam, sam, bed, gff3, cvf, mC, simpleText (default)].

--debug : [value : = number of lines] output limit number of lines from input for testing.

-r,--reverse : reverse strand + to - and verse versa.

-s,--string : vaule : Simple (default), MySQL, or MySQL0 (old).

-f,--force : force to overwrite all outputs.

-a,--algorithm : value = one of 'Sum', 'Coverage'(default), or 'Count'.

--class : assigned a class value for converting bed file to Simple file.

--class_conversion : add new classes or convert old classes to new ones for mC.

--tmp : temporary working directory. default : '/tmp'

--samtools : samtools path.

-k,--keep : value = keepSimple[,keepSmall] for keeping the intermediate Simple files, small Exons.

-q,--quiet : quiet.

-h,--help : print out help().

C. Set Default Track Parameters:

php simple_db.php -t SimpleSettings -o Destination_directory [ --user ... [ ... ]]

1) user : set read permission for users only. default : all. Example: --user hchen,hchen2
2) group : set read permission for groups only. default : none if user is set.
3) title
4) info
5) name : set default track name.
6) path : set default path.
7) scale
8) color
9) single
10) policy
11) height : set track height.
12) class : set custom class. for example: '-CG CHG CAC CHH'.
13) boxHeight : set display box height (minimum value : 1 (px)).
14) boxWidthMin : set boxWidthMin (px) to show small childs. Default 0 - not show.
15) provider_name : set provider's name.
16) provider_email : set provider's email.
17) institution_name : set provider's institution.
18) institution_url : set provider's institution link url.
19) institution_logo : set provider's institution logo url.

You can also use command line : > echo $value > $dest/.$attr to set attr = value.

D. Convert custom format data to SimpleText file:

php simple_format.php -t ... -i ... -o Simple.txt -d ...

1) gff3 -> SimpleModels : php simple_format.php -t SimpleModels -i src.gff3 -o out.txt -d gff
2) bam -> SimpleReads : php simple_format.php -t SimpleReads -i src.bam -o out.txt -d bam
3) sam -> SimpleReads : php simple_format.php -t SimpleReads -i src.sam -o out.txt -d sam
4) mC -> SimpleMeth : php simple_format.php -t SimpleMeth -i src.allc -o out.txt -d mC
5.1) bed -> SimpleModels : php simple_format.php -t SimpleModels -i src.bed -o out.txt -d bed
5.2) bed -> SimpleIntensity : php simple_format.php -t SimpleIntensity -i src.bed -o out.txt -d bed
7) vcf -> SimpleSNPs : php simple_format.php -t SimpleSNPs -i src.vcf -o out.txt -d vcf

3. MySQL Database [optional]

The genomic data used for track display could be stored in mySQL database and fetched by common mySQL modules.

A. Gene models or repeat mask data

1) Data type : Gene model list, repeat mask list, or bed list
2) mySQL table attributes: [parent], [id], assembly, [strand], [class], start, end, and [description]
3) Track type : ModelsTrack or MaskTrack
4) Fetcher driver : common_models.php or custom_models.php
5) MySQL query : select parent, id, strand, class, start, end-start+1 from table where assembly= ...
6) Options for using custom_models.php : "parent:,id:,strand:+,class:read"

B. Single reads or small reads

1) Data type : Single read list or small RNA read list
2) mySQL table attributes: id, assembly, start, end, strand, and sequence, [copies], [count]
3) Track type : ReadsTrack or SmallReadsTrack
4) Fetcher driver : common_reads.php, custom_reads.php or common_smrna.php
5) MySQL query : select id, start, end, strand, sequence, 1, 1 from table where ...
6) options for using custom_reads.php : class:read;strand:+;count:places;copies:copies;sequence=sequenceA

C. PairedEnd reads or mRNA reads

1) Data type : PairedEnd read list, or mRNA sequence reads
2) mySQL table attributes: id, assembly, start, end, strand, sequenceA, and sequenceB, [copies], [count]
3) Track type : PairedEndTrack
4) Fetcher driver : common_pe.php
5) MySQL query : select id, strand, start, end, length(sequenceA), length(sequenceB), sequenceA, sequenceB from table where ...

D. Methylation data

1) Data type : Methylation calls
2) mySQL table attributes : [id], assembly, position, class, strand, mc, h
3) Track type : MethTrack
4) Fetcher driver : common_wgta.php
5) MySQL query : select position, class, strand, mc, h from table where

Note: [optional] = optional attributes.

4. Anno-J Browser Tracks

A. ModelsTrack

1) Data type : ModelsList
2) Canvas type : ModelsCanvas
3) Fetcher driver : common_models.php, custom_models.php or simple_models.php
4) Properties : path, height, scale, single, iconCls, showControls, showLabels, showArrows, boxHeight, boxWidthMin, alignControl, showWalks, policy

B. MaskTrack

1) Data type : ModelsList
2) Canvas type : MaskCanvas
3) Fetcher driver : common_models.php, custom_models.php, simple_mask.php or simple_models.php
4) Properties : path, height, single, iconCls, showControls, boxHeight, showWalks, policy

C. ReadsTrack

1) Data type : HistogramList (>=10:1), ReadsList (<10:1)
2) Canvas type : HistogramCanvas, ReadsCanvas
3) Fetcher driver : common_reads.php, custom_reads.php, or simple_reads.php
4) Properties : path, height, scale, single, iconCls, showControls, color, showWalks, policy

D. SmallReadsTrack

1) Data type : HistogramList (>=10:1), SmallReadsList (<10:1)
2) Canvas type : HistogramCanvas, ReadsCanvas
3) Fetcher driver : common_smrna.php, or simple_smrna.php
4) Properties : path, height, scale, single, iconCls, showControls, color, showWalks, policy

E. PairedEndTrack

1) Data type : HistogramList (>=10:1), PairedEndList (<10:1)
2) Canvas type : HistogramCanvas, PairedEndCanvas
3) Fetcher driver : common_pe.php or simple_pe.php
4) Properties : path, height, scale, single, iconCls, showControls, color, showWalks, policy

F. IntensityTrack, MicroarrayTrack

1) Data type : HistogramList
2) Canvas type : HistogramCanvas
3) Fetcher driver : simple_intensity.php or simple_microarray.php
4) Properties : path, height, scale, single, iconCls, showControls, color, showWalks, policy

G. MethTrack

1) Data type : HistogramList
2) Canvas type : HistogramCanvas
3) Fetcher driver : common_wtga.php or simple_wtga.php
4) Properties : path, height, scale, single, iconCls, showControls, color, showWalks, policy, class

H. AlignsTrack

1) Data type : ModelsList
2) Canvas type : AlignsCanvas
3) Fetcher driver : simple_aligns.php
4) Properties : path, height, single, iconCls, showControls, boxHeight, alignControl, policy

I. SNPsTrack

1) Data type : ReadsList
2) Canvas type : SNPsCanvas
3) Fetcher driver : simple_snps.php
4) Properties : path, height, single, iconCls, showControls, boxHeight, boxWidthMin, showWalks, policy

J. HiCTrack

5. Screen videos

A. Anno-J browser tutorial

B. jBuilder tutorial

C. SimpleDB tutorial