List file names with modified time stamp customized

Commands used to list files from a directory yields results of more details and time stamp with GMT values. To customize the output to display the file names only with time stamp in custom format with sorted order, use:

$ find . -mindepth 1 -maxdepth 1 -type f -printf "%TY-%Tm-%Td %TH:%TM:%.2TS %f\n" | sort -n 

Here the mindepth and maxdepth parameters were used to use the current directory for the search. That can be modified as per need. And the output will be,

2014-04-22 19:34:36 .bash_logout
2014-04-22 19:34:36 examples.desktop
2014-04-22 19:34:36 .profile
2014-04-22 21:26:24 dropcaches.sh
2014-04-22 22:09:20 .bashrc
2014-05-04 08:41:42 .dmrc

Bulk rename using bash rename command

Rename command with RegEx pattern can be used to do the bulk rename operation. For example, to append the ip address of the machine in between the log files supplied, the below code is used.

#!/bin/bash
ip=$(hostname -I | sed 's/[^0-9.]*//g')
file="_"$ip"."
rename -v "s/(\w+).(\w+)+/\$1$file\$2/" *.log*

The output will like:

shell.log renamed as shell_192.168.1.2.log
shell.log.1 renamed as shell_192.168.1.2.log.1
shell.log.2 renamed as shell_192.168.1.2.log.2
shell.log.zip renamed as shell_192.168.1.2.log.zip

Apache authentication and authorization using LDAPS

Apache web Authentication using LDAP/LDAPS requires two steps.

1. Enabling public key at client side
2. Configure LDAPS server in Apache (assuming that your LDAPS server is already running)

To enable secure connection with LDAPS server, the public key should be installed at Apache machine server follow below steps

1. Copy the key in to /etc/ldap/cacerts or /etc/openldap/cacerts
2. Configure the key entry in /etc/ldap.conf

To configure Apache server to communicate LDAPS server for authentication, add this into Apache http.conf or default.conf file.

LDAPTrustedGlobalCert CA_BASE64 /etc/openldap/cacerts/ldap_pubkey.pem
<Directory /var/www/html/>
    AuthName "Apache authentication using Ldaps Server"
    AuthType Basic
    AuthBasicProvider ldap
    AuthzLDAPAuthoritative off
    AuthLDAPURL ldaps://ldaps.test.com:636/ou=users,dc=test,dc=com?uid
    AuthLDAPBindDN cn=manager,dc=test,dc=com
    AuthLDAPBindPassword <pwd>
    #Allowed user list
    Require ldap-user user1 user2

    Options Indexes FollowSymLinks
    AllowOverride All
    Order allow,deny
    Allow from all
</Directory>

Err: Client denied by server configuration after Apache upgrade

From Apache 2.4, syntax of user directive options were changed. So that Apache configuration from old version to new 2.4 or later will cause this error due the incorrect syntax.

# Apache version 2.2 or lower configuration:
<Directory /docroot>
Order allow,deny
Allow from all
</Directory>

# Apacheversion 2.4 or later configuration:
<Directory /docroot>
Require all granted
</Directory>

For more information on new syntax visit Apache Access Directive in Details.

So use latest directory access directives syntax to avoid the same type of errors.

PDOException: Invalid text representation

Error:
The website encountered an unexpected error. Please try again later. [PDOException: SQLSTATE[22P02]: Invalid text representation: 7 ERROR: invalid input syntax for integer: "as" LINE 5: WHERE (base.nid IN ('as')) ^: ..... (line 191 of/var/www/html/drupal/includes/entity.inc).]

Solution: 
Drupal throws PDO exception error like above while giving url patterns like,

localhost/drupal/node/abc
localhost/drupal/node/1.
localhost/drupal/node/1.23
localhost/drupal/node/<any special characters like *&^%$#@!>

This because of invalid input for select query while it is expecting integer. This is core level bug and the wrong input needs to be filtered in entity.inc of core module.

//add the below lines inside function load() in entity.inc.
//Patch to avoid non numeric node id's which causes pdo exceptions
if (is_array($ids)) {
        // Remove all non numeric ids.
        $ids = array_filter($ids, function($x){
                return preg_match('/^[0-9]+$/', $x);
        });
}

Split PDF into multiple with n number of pages each

The following Linux shell script splits the source PDF file into multiple files with n pages of each. Totall no of pages in given input file calculated automatically and user needs to give the pages wants in each output file. This can be configured using the variable step. Default it splits the files with 5 pages.

#!/bin/bash
file="src/test.pdf";
#count total no of pages in the file
total_pages=$(pdfinfo $file | grep Pages | awk '{print $2}')
from=1
#step defines how many pages should be in output files
step=5;
while [ $from -lt $total_pages ]
do
        to=$((from+step-1));
        if [ $to -gt $total_pages ]; then
                to=$total_pages;
        fi
        echo "Generating pdf file : pages-$from-$to.pdf...";
        pdftk $file cat $from-$to output "test-$from.pdf";
        from=$((to+1));
        echo "Done.";
done

Drop virtual memory cache from Linux box to free RAM space

In performance perspective mode, Linux box is designed to keep the frequently used application's data in memory to boost up system performance. System with low RAM space will affected by opposite manner that the cache occupies RAM space mostly and system got slow even got hanged also. Apart from theory, we need to resolve that to run the machine without hang.

To free such cache from RAM, run the below command often. This will erase the cache with simple value what we give.

$ echo 3 > /proc/sys/vm/drop_caches

Remove line numbers from source code using sed

While getting sample code from internet, it may comes with line number. To simply remove those lines from the code file use the below ways.

Using sed command with RegEx pattern in Linux, that can be achieved.

To remove it from console:

$ sed -i "s/^ *[0-9.]* *//g" code.txt

To remove it from vim interface:

:%s/^ *[0-9.]* *//g

And the same RegEx pattern can be used in anywhere to remove the line numbers from the source code file.

 

Ubuntu upgrading - mount /tmp as noexec error

Error:

     Error : mount /tmp as noexec error

Reason:

While upgrading Ubuntu it stores the temporary files in temp directory and tries to execute from the same location. By default, /tmp directory doesn't contain execute permission for security reasons. So that it can't be start from the location /tmp.

Solution:

For while, give execute permission to /tmp and run the upgrade once.

Run the below command to do the same.

    mount -o remount exec /tmp

Port 80 is already used by another program in windows

This may happens in Windows machines while installing Apache.

This because of the port may be used by some other program like Skype or default os service. To resolve this try the below ways:

• Disable HTTP.sys driver that was started by some of the Windows inbuilt applications
• Disable the HTTP in registry under the path "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP"

Refresh system settings from sysctl.conf without restart Linux

The custom settings entered in sysctl.conf can be easily revoked without restarting the Linux box. Using sysctl command with -p argument refreshes the machine with new settings.

$ sysctl -p

Turn USB write protect flag off

Error:

[sdb] No Caching mode page present
[sdb] Assuming drive cache: write through
[sdb] Attached SCSI removable disk

Reason:

Most of the virus programs try to turn on write protect flag of USB disks after inserting their malicious code programs/applications in to that. So that none of the programs can able to delete that even format the disk too.

Solution:

Take a Linux console and give the below command to reset the write protect flag. Then the drive comes to normal read/write mode.

Assuming that your USB device is /dev/sdb.

         hdparm -r0 /dev/sdb

Finding last occurrence of a label and its value in Spreadsheet

	A	B	C	D	E	F
1		A	11		A	19
2		B	12		B	16
3		C	13		C	15
4		D	14		D	18
5		C	15
6		B	16
7		A	17
8		D	18
9		A	19

Finding a value of label that occurs as last in a large list is always complex one. Spreadsheet's lookup functions will helps up to some level but not completely. Using the following workaround, its possible and the desired value can be populated.

Feed the below values in respective cells and press CTRL+SHIFT+ENTER to run the script.

F1 = INDEX($C$1:$C$9,MAX(ROW($A$1:$A$9)*($B$1:$B$9=$E1)))
F2 = INDEX($C$1:$C$9,MAX(ROW($A$1:$A$9)*($B$1:$B$9=$E2)))
F3 = INDEX($C$1:$C$9,MAX(ROW($A$1:$A$9)*($B$1:$B$9=$E3)))
F4 = INDEX($C$1:$C$9,MAX(ROW($A$1:$A$9)*($B$1:$B$9=$E4)))

Here Column "A" was referred just for row number reference nothing else. Column "A" may have any value or empty.

Note: Labels are A-D and values are numeric values.

Table row into Column using unnest in PostgreSQL

PostgreSQL result rows can be converted to Columns using Array function called unnest.

For example, the below query result

uid | name | mail
---------------------------------
01 | admin | admin@test.com

can be converted like below.

ColumnName | Value
---------------------------------
UID         | 01
Name | admin
Mail         | admin@test.com
----------         | ----------
Total Count | 1

Use this query to convert row values into columns:

--unnest to give array as rows
WITH
   x AS (
SELECT *
FROM   users t where uid > 0
   ),
   y AS (
SELECT ARRAY [uid::varchar, name, mail] AS val,
ARRAY ['UID','Name','Mail'] AS item
FROM   x
   )
SELECT unnest(item) AS ColumnName,
unnest(val) AS Value
FROM   x,y
UNION  ALL
SELECT '----------'::text, '----------'::text
UNION  ALL
SELECT 'Total Count'::text, count(x)::text from x

Drupal PDO error : Invalid input syntax for integer

Error:

        PDOException: SQLSTATE[22P02]: Invalid text representation: 7 ERROR: invalid input syntax for integer:

Reason:

Drupal will not recognize the node URL when the node id is not an integer. This will happens when URL like node/abc instead of node/12. It fails to get the desired node URL location from the database. It's bug of Drupal Core.

Solution:

To filter out the non numeric node id's from the URL from executing use the below patch in Drupal core file called includes/entity.inc. Add the lines as first in function called load().

    public function load($ids = array(), $conditions = array()) {
        //Add the below code patch

        //-----Patch Start----

        if (is_array($ids)) {
        // Removes all non numeric ids.
        $ids = array_filter($ids, function($x){
                return preg_match('/^[0-9]+$/', $x);
                });

        }

        //-----Patch End----

        $entities = array();

Clear sudo password remember in linux

Normally Linux will keep the session of sudo command in terminal for a while and it will not ask for password again when try to use the sudo before the expiry time. You can resolve this by clearing the session immediately by giving the below command.

sudo -k

Custom access denied error page for sites/default/files url in Apache

User will get access denied page after setting up standard codes in htaccess file of site/default/files folder. This error page can be customized by adding our used defined html page to that.

The page can be showed by redirecting from access denied default page by adding the below line in htaccess file.

ErrorDocument 404 /error/HTTP_NOT_FOUND.html.var

It redirects and shows the page from Apache's default error page where user either can modify the code of that file or add new custom html file.

Compile shell scripts to encrypted binaries using SHC

Shell scripts are the plain code files and it could be modified easily when in the case of distributing. To avoid that the scripts can be converted to some other format like binary. Using SHC library the Unix shell scritps can be converted to encrypted binaries so that reverse engineering of that is not that much easy.

The Debian or Tar source of SHC are availble in Open Source repositories. Install the tool and give the below command to make the object file.

$ shc -v -r -T -f test.sh
$ mv test.sh.x test
$ ./test 

This will output one object file and one c file. Just rename output file with the suffix ".x" (ie.test.sh.x) to some name and use it anyware.

Bulk Convert Office Documents into PDF using unoconv

There are many methods availabel to convert bulk amount of document files into PDF files. Using Using Open Source libarary/tool we can able to do it programatically. (ie. Open Office headless service, unoconv and etc.,).

The drawback of using Open Office headless service in the case of files are in sub directories too is it will output the files in current working directory instead of actual source file path. In that case we can use unoconv tool. It supports all the formats that Open Office supports.

Unoconv

       Unoconv is a command line utility that can convert any file format that LibreOffice can import, to any file format that LibreOffice is capable of exporting. To know what are the types it is supporting,

$ unoconv --show

Example

       To convert the list odt files from path and its sub directories use the below code. This can be modified to different input and output formats as required.

#!/bin/bash
#To avoid errors due to spaces in file names
IFS="$(printf '\n\t')"
#Finds odt files in current directory and its sub directories and process one by one
for file in $(find . -name "*.odt" -type f); do
  echo "Processing File : $file ...";
  unoconv -d document -f pdf "$file";
Done

Internal field separator (IFS) for shell scripts

Internal field separator (IFS)

By default, Unix/Linux shell scripts takes space as field separator to process input text. This may work in some scenarios but not all case. By setting IFS variable to some character(s) will take care of input parsing using that.

For example, while looping through the directory file names in Linux, spaces in file name will divert the actual outcome what you are expecting. In that case use the below variable to set new line and tab characters to be a field separator and done it.

   IFS="$(printf '\n\t')"
   for fileItem in ./*
   do
        # do your stuff ...
   done