MASH commands

Following is the list of mash commands used to manipulate data within Ampool.

Command syntax and options

create table

create memory-table.

Note

Please refer the following table for valid command line options that are supported since v1.2 release.

  • x - Supported
  • N/A - Invalid/Not Applicable.
  • N/S - Not Supported in current release.
Operation/Attribute MTable FTable
create table x x
name x x
type ORDERED_VERSIONED, UNORDERED IMMUTABLE
disk-persistence true/false true
eviction-policy OVERFLOW_TO_DISK, NO_ACTION OVERFLOW_TO_TIER
disk-store x x
disk-write-policy Sync/Async Sync/Async
expiration-timeout N/S N/S
expiration-action N/S N/S
ncopies x x
nbuckets x x
versions x 1
columns x x
schema-json x x
block-size N/A x
local-max-memory x x
total-max-memory x x
tier-stores NA Comma separated list of tier store(s)

Syntax:

create table --name=value --type=value [--columns=value(,value)] [--schema-json=value]
[--ncopies=value\] [--versions=value\] [--disk-persistence=value]
[--disk-write-policy=value\] [--disk-store=value] [--nbuckets=value] [--eviction-policy=value]
[--expiration-timeout=value] [--expiration-action=value] [--block-size=value]
[--local-max-memory=value] [--total-max-memory=value] [--tier-stores=value]
[--tier1-time-to-expire=value] [--tier2-time-to-expire=value]
[--tier1-time-partition-interval=value] [--tier2-time-partition-interval=value]

Parameters:

Name Description Default Value
--name Required. Name of the memory table to be created. Valid name includes only alphanumeric chars [A-Za-z0-9_-]
--type Required. Type of the memory table.
--columns Names of columns to be added. A comma separated list of column names e.g Age,Salary, etc. Don't use quotes while specifying column names.
--schema-json The schema of the table to be created. A valid JSON with column-name and respective string representation of column-type.
--ncopies Sets the Redundant copies of the data for a table.Please choose a value between 0 and 3 (inclusive) 1
--versions Set the Maximum number of versions to be kept in memory for versioning. Once the max limit is reached oldest version of that row will be deleted default max version is 1. 1
--disk-persistence Enables disk persistence for table. false
--disk-write-policy Disk write policy used for disk persistence of table. SYNCHRONOUS
--disk-store Disk Store to be used by this table. "list disk-stores" can be used to display existing disk stores.
--nbuckets The maximum number buckets/partitions to be created for the table based on the specified partitioning scheme. 113
--eviction-policy The eviction policy to be used when the heap space runs low. For tables with IMMUTABLE type it is always OVERFLOW_TO_TIER.
--expiration-timeout Time in seconds after which a row will get expired.
--expiration-action The action that should take place when a row expires.
--local-max-memory The maximum amount of memory, in megabytes, to be used by the table on a server. 90% of available heap
--total-max-memory The maximum amount of memory, in megabytes, to be used by the table on all servers.
--block-size IMMUTABLE The maximum number of rows to be stored as a single block in a single key-value entry. 1000
--tier-stores IMMUTABLE The comma separated list of tier-stores to be used by this table in the same order. \"list tier-stores\" can be used to display existing tier stores. DefaultLocalORCStore
--tier1-time-to-expire Time in hours for first tier after which a row will get expired. None
--tier2-time-to-expire Time in hours for second tier after which a row will get expired. None
--tier1-time-partition-interval Time in hours for first tier to be used for time partitoning the data. 1 Hour
--tier2-time-partition-interval Time in hours for second tier to be used for time partitoning the data. 1 Hour

Note

When creating a memory table either --columns or --schema-json option must be provided.

Example Commands :

create table --name=MTable --type=ORDERED_VERSIONED –columns=c1,c2
create table --name=MTable1 --type=ORDERED_VERSIONED --columns=c1,c2 --ncopies=1 --versions=1 --disk-persistence=true –disk-write-policy=ASYNCHRONOUS
create table --name=ImmutableTable --type=IMMUTABLE --columns=ID,NAME
create table --name=TableWithTypes --type=IMMUTABLE --schema-json="{'ID':'INT','NAME':'STRING'}"
create table --name=TableWithTiers --type=IMMUTABLE --schema-json="{'ID':'INT','NAME':'STRING'}" --tier-stores=DefaultLocalORCStore,HdfsStore
create table --name=TableWithTiers --type=IMMUTABLE --schema-json="{'ID':'INT','NAME':'STRING'}" --tier-stores=DefaultLocalORCStore,HdfsStore --tier1-time-to-expire=24 --tier2-time-partition-interval=24

Sample Output:

Successfully created the memory table: <table-name>

delete table

delete memory-table

Syntax:

delete table --name=value

Parameters:

Name Description Default Value
--name Required. Name of the memory table to be created. Valid name includes only alphanumeric chars [A-Za-z0-9_-]

Example Commands:

delete table --name=/MTable

Sample Output:

Successfully deleted the memory table: "/MTable"

describe table

describe memory-table

Syntax:

describe table --name=value

Parameters:

Name Description Default Value
--name Required. Name of the memory table to be created. Valid name includes only alphanumeric chars [A-Za-z0-9_-]

Example Commands:

describe table --name=/MTable1

Sample Output:

................................................................................
Name            : MTable1
Data Policy     : persistent partition
Hosting Members : s1

Non-Default Attributes Shared By Hosting Members

      Type        |          Name           | Value
----------------- | ------------------------|----------------------------
Table             | data-policy             | PERSISTENT_PARTITION
                  | disk-store-name         | AMPOOL_DEFAULT_DISK_STORE
                  | size                    | 0
Eviction          | eviction-action         | overflow-to-disk
                  | eviction-algorithm      | lru-heap-percentage
Partition         | redundant-copies        | 1
                  | partition-resolver      | io.ampool.monarch.table.functions.MTableRangePartitionResolver
MTable Attributes | Column 0                | c1
                  | Column 1                | c2
                  | Disk Write Policy       | ASYNCHRONOUS
                  | Max Versions            | 1
                  | Persistence Enabled     | true
                  | Redundant Configuration | 1
                  | Table Type              | ORDERED_VERSIONED
                  | Total Number Of Splits  | 113

list tables

Display list of tables.

Syntax:

list tables [--group=value] [--member=value]

Parameters:

Name Description Default Value
--group Group of members for which tables will be displayed.
--member Name/Id of the member for which tables will be displayed.

Example Commands:

list tables
list tables  --member=s1

Sample Output:

     Name      | Type
-------------- | -----------------
MTable         | Ordered Versioned
MTable1        | Ordered Versioned
ImmutableTable | Immutable

tput

Add/Update an entry in a memory-table.  

Syntax:

tput --key=value --value=value --table=value

Parameters:

Name Description Default Value
--key Required. String or JSON text from which to create the key. Examples include: "James", "100L" and "('id': 'l34s')".
--value Required. Value is required for parameter "value: Comma separated key=value pairs of column values". Example: colum1=value1,column2=value2,column3=value3
--table Required. MTable name into which the entry will be put.

Example Commands:

tput --key=1 --value=c1=1,c2=2 --table=/MTable1

Sample Output:

MPut completed successfully

tget

Display an entry in memory-table.

Syntax:

tget --key=value --table=value

Parameters:

Name Description Default Value
--key Required. String or JSON text from which to create the key. Examples include: "James", "100L" and "('id': 'l34s')".
--table Required. MTable name from which the entry will be retrieved.

Example Commands:

tget --key=1 --table=/MTable1

Sample Output:

KEY : 1
c1  : 1
c2  : 2
-----------

tscan

Display the entries from the memory-table.

Syntax:

tscan --table=value [--startkey=value] [--stopkey=value]
[--key-type=value] [--max-limit=value] [--step-name=value]
[--interactive=value]

Parameters:

Name Description Default Value
--table Required. MTable name from which the entries will be retrieved.
--stopkey String or JSON text from which to create the key. Examples include: "James", "100L" and "('id': 'l34s')".
--table Required. MTable name from which the entries will be retrieved.
--key-type The type to be used to convert start/stop key to byte-array. Default is STRING. Using BINARY will convert comma separated bytes into byte-array. STRING
--max-limit Max number of records to be retrieved. 1000
--interactive Whether or not this query is interactive. If false then all results will be displayed at once. false

Example Commands:

tscan --table=/MTable1

tscan --table=/MTable1 --startkey=1
tscan --table=/MTable1 --startkey=1 --stopkey=2
tscan --table=/MTable1 --startkey=1 --stopkey=2 --key-type=STRING
tscan --table=/MTable1 --startkey=1 --stopkey=2 --key-type=STRING --max-limit=10
tscan --table=/MTable1 --startkey=1 --stopkey=2 --key-type=STRING --max-limit=10 –interactive=false

Sample Output:

Result     : true
startCount : 0
endCount   : 20
Rows       : 1

Result
------------------------------
RowID = [49] c1=[49], c2=[50]

NEXT_STEP_NAME : END

tappend

Append entries in FTable.

Syntax:

tappend --table=value [--value-json=value]

Parameters:

Name Description Default Value
--table Required. FTable name into which the entries will be appended.
--value-json Required. JSON text from which to create the row

Example Commands:

tappend --table=/ImmutableTable --value-json="{'ID': '11', 'NAME': 'PI'}"

Sample Output:

Append on FTable completed successfully

create tier-store

Creates tier store to archive data from Immutable table. Ampool support both shared as well as local store with multiple file formats. With following command user can configure tier store and attach while creating table.

Syntax:

create tier-store --name=value --handler=value --reader=value --writer=value [--props-file=<path to store properties file>] [--reader-opts=<path to reader properties file>] [--writer-opts=<path to writer properties file>]

Parameters:

Name Description Default Value
--name Required. Tier store name that will used to attach to the immutable table.
--handler Required. Fully Qualified Class Name for tier store. This class must be in the classpath of every ampool server
--reader Required. Fully Qualified Class Name for tier store reader implementation that store will use to read. This class must be in the classpath of every ampool server
--writer Required. Fully Qualified Class Name for tier store writer implementation that store will use to write. This class must be in the classpath of every ampool server
--props-file Absolute path to the properties file required by store. This file must available at the same path on every ampool server
--reader-opts Absolute path to the properties file required by reader. This file must available at the same path on every ampool server
--writer-opts Absolute path to the properties file required by writer. This file must available at the same path on every ampool server

Example Commands:

create tier-store --name=LocalStore --handler=io.ampool.tierstore.stores.LocalTierStore --reader=io.ampool.tierstore.readers.orc.TierStoreORCReader --writer=io.ampool.tierstore.writers.orc.TierStoreORCWriter --props-file=storeProps.properties --reader-opts=readerProps.properties --writer-opts=writerProps.properties

Sample Output:

Successfully created tier store: LocalStore

list tier-stores

Display list of tier-stores.

Syntax:

list tier-stores [--group=value] [--member=value]

Parameters:

Name Description Default Value
--group Group of members for which tier-stores will be displayed.
--member Name/Id of the member for which tier-stores will be displayed.

Example Commands:

list tier-stores
list tier-stores  --member=s1

Sample Output:

Name
--------------------
HDFSBasedORCStore
DefaultLocalORCStore

describe tier-store

Describes tier-store

Syntax:

describe tier-store --name=value

Parameters:

Name Description Default Value
--name Required. Name of the tier-store.

Example Commands:

describe tier-store --name=DefaultLocalORCStore

Sample Output:

..........................................................................
Name          : DefaultLocalORCStore
Handler class : io.ampool.tierstore.stores.LocalTierStore
Reader class  : io.ampool.tierstore.readers.orc.TierStoreORCReader
Writer class  : io.ampool.tierstore.writers.orc.TierStoreORCWriter

Properties

      Type        |              Name               | Value
----------------- | ------------------------------- | ------------
Store Properties  | partitioning.interval.ns        | 600000000000
                  | base.dir.path                   | ./ORC
                  | time.based.partitioning.enabled | true
Writer Properties | stripe.size                     | 1000
                  | new.index.stride                | 1000
                  | compression.kind                | NONE
                  | buffer.size                     | 1000

Initializing default store packaged by ampool

Ampool package comes with shared and non-shared store implementations to use. Ampool provides reader and writer (ORC format) to archive the Immutable table data.

The local disk based tier store is instantiated with the ampool server. The name of the default tier store is DefaultLocalORCStore. This archives data into the /ORC directory in ORC format. If an immutable table with eviction-policy as "OVERFLOW_TO_TIER" is created without explicitly attaching stores (using a --tier-stores argument in create table command) then DefaultLocalORCStore is attached as archival tier store.

Using following steps we can instantiate following shared stores to archive data.

1. HDFS Store

Command:

create tier-store --name=HDFSBasedORCStore --handler=io.ampool.tierstore.stores.HDFSTierStore --reader=io.ampool.tierstore.readers.orc.TierStoreORCReader --writer=io.ampool.tierstore.writers.orc.TierStoreORCWriter --props-file=storeProps.properties

Store properties:

storeProps.properties has following configurations

Name Value Description
base.dir.path ORC Base path to archive data
hdfs.site.xml.path /etc/hadoop/hdfs-site.xml Absolute path to hdfs-site.xml, a hdfs configuration file
hadoop.site.xml.path /etc/hadoop/core-site.xml Absolute path to core-site.xml, a hadoop common configuration file
time.based.partitioning.enabled true Whether to enable time based partitioning while archiving data based on insertion timestamp for record
partitioning.interval.ns 600000000000 Interval in nano-seconds to create new time partition
security.user.name user principal user principal used for writing/reading to/from HDFS
security.keytab.path keytab path keytab path for the user. Path should be accessible on Ampool server

Sample Output:

Successfully created tier store: HDFSBasedORCStore

2. S3 Store

Command:

create tier-store --name=S3ORCStore --handler=io.ampool.tierstore.stores.S3TierStore --reader=io.ampool.tierstore.readers.orc.TierStoreORCReader --writer=io.ampool.tierstore.writers.orc.TierStoreORCWriter --props-file=storeProps.properties

Store properties:

storeProps.properties has following configurations

Name Value Description
base.dir.path ORC Base path to archive data
fs.s3a.access.key Access key used for s3 access
fs.s3a.secret.key Secret key used for s3 access
time.based.partitioning.enabled true Whether to enable time based partitioning while archiving data based on insertion timestamp for record
partitioning.interval.ns 600000000000 Interval in nano-seconds to create new time partition
hadoop.site.xml.path Absolute path to configuration file
Configuration file
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>s3a://<S3 bucket name></value>
  </property>
</configuration>

Sample Output:

Successfully created tier store: S3ORCStore

Available file formats

Ampool package provides support for following file formats. Use respective value as --reader and --writer option while creating tier store.

1. ORC - Optimized Row Columnar

Ampool provides support to store data in ORC file format which most commonly used to store hive data. User can use following reader and writer class while creating tier store to instantiate ORC based store.

Reader : Reader implementation for Tier store

Ampool packages io.ampool.tierstore.readers.orc.TierStoreORCReader class. This class is capable of reading ORC files.

Writer : Writer implementation for Tier store

Ampool packages io.ampool.tierstore.writers.orc.TierStoreORCWriter class. This class is capable of writing files in ORC format.

User can pass following properties to configure ORC file format as per there needs.

Property Description Default Value Values supported
compression.kind Compression algorithms that can be applied to ORC files. NONE NONE, ZLIB, SNAPPY, LZO
stripe.size Number of bytes per stripe. 1000
buffer.size Number to buffer and compress at once. 1000
new.index.stride Number of rows between row index entries or 0 to suppress all indexes 1000

This reader and writer can be used with non-shared as well as shared tier store.

2. Parquet - Columnar storage format

Ampool provides support to store data in Parquet file format. User can use following reader and writer class while creating tier store to instantiate Parquet based store.

Reader : Reader implementation for Tier store

Ampool packages io.ampool.tierstore.readers.parquet.TierStoreParquetReader class. This class is capable of reading Parquet files.

Writer : Writer implementation for Tier store

Ampool packages io.ampool.tierstore.writers.parquet.TierStoreParquetWriter class. This class is capable of writing files in Parquet format.

User can pass following properties to configure Parquet file format as per there needs.

Property Description Default Value Values supported
compression.codec.name Compression algorithms to be used to compress. UNCOMPRESSED UNCOMPRESSED, GZIP, SNAPPY, LZO
block.size Threshold of block size in bytes 256 * 1024 * 1024 (256 MB))
page.size Size of page in bytes. Blocks are subdivided into blocks. 64 * 1024 (64 KB)

This reader and writer can be used with non-shared as well as shared tier store.