parsing - Parse JQ output through external bash function?


I want to parse out data out of a log file which consist of JSON sting and I wonder if there's a way for me to use a bash function to perform any custom parsing instead of overloading jq command.


tail errors.log --follow | jq --raw-output '. | [.server_name, .server_port, .request_file] | @tsv' 

Outputs:     80     /var/www/

I want to parse 3rd column to cut the string to exclude /var/www/ part where /var/www/ is the document root, and /var/www/ is the public html section of the site. Therefore I would like to leave my output as /subdomain/public (or from the example /www/public).

I wonder if I can somehow inject a bash function to parse .request_file column? Or how would I do that using jq?

I'm having issues piping out the output of any part of this command that would allow me to do any sort of string manipulation.

1 Answer: 

Use a BashFAQ #1 while read loop to iterate over the lines, and a BashFAQ #100 parameter expansion to perform the desired modifications:

tail -f -- errors.log \
  | jq --raw-output --unbuffered \
       '[.server_name, .server_port, .request_file] | @tsv' \
  | while IFS=$'\t' read -r server_name server_port request_file; do
      printf '%s\t%s\t%s\n' "$server_name" "$server_port" "/${request_file#/var/www/*/}"

Note the use of --unbuffered, to force jq to flush its output lines immediately rather than buffering them. This has a performance penalty (so it's not default), but it ensures that you get output immediately when reading from a potentially-slow input source.

That said, it's also easy to remove a prefix in jq, so there's no particular reason to do the above:

tail -f -- errors.log | jq -r '
  def withoutPrefix: sub("^([/][^/]+){3}"; "");
  [.server_name, .server_port, (.request_file | withoutPrefix)] | @tsv'

More Articles

php - mcrypt module open function is missing in php7.2.7

In PHP5 I was able to use mcrypt_module_open and now in PHP7.2.7 (which I just upgrade to) the function is missing and while searching, I discovered that it has been replaced by OpenSSL. My question is this: What is the equivalent of mcrypt_module_open in OPenSSL?

android - How to return different code and error with MockRestAdapter

I use MockRestAdapter to return mock data in my tests, but I'd also like to test errors (401, 503, UnknownHostException, etc)For SocketTimeoutException, there's an API, but how about different response code?I've tried MockWebServer but no matter what I enqueue, I always get a 200 with the mock data

rust - How do I move multiple values out of a boxed value without calling clone?

This question already has an answer here: How to bind multiple fields of a boxed struct without getting “use moved value” error? 2 answers

c++ - Boost.Asio SSL thread safety

Do I create one strand that all of my SSL sockets share, or one strand per SSL context (shared by any associated sockets)?Boost.Asio SSL documentation states this, but it doesn't mention contexts. I assume that this means I must use only one strand for everything, but I think this was written before

data structures - Is there any practical usage of Doubly Linked List, Queues and Stacks?

I've been coding for quite sometime now. And my work pertains to solving real-world business scenarios. However, I have not really come across any practical usage of some of the data structures like the Linked List, Queues and Stacks etc.Not even at the business framework level. Of course, there is

Know of any setup tutorials for SQL Server Express and C#?

I'm a C# and MySQL developer, but I'm looking into Microsoft's SQL Server for a new project. I'm familiar with MySQL syntax and the .NET connector. Does anyone know of any decent tutorials that just cover the code to interact with SQL Server? No C# basics or SQL basics, just how to get the two tec

libgit2sharp: how to set multiple values with same key

how do I set multiple values for a config-key? Some sections supports multiple values:[remote "origin"] url = fetch = +refs/heads/master:refs/remotes/origin/master fetch = +refs/heads/qa/*:refs/remotes/origin/qa/*Something like this is not working in

r - split string at every character

I want to split every character in a string and output it as comma or tab separated characters:I need to use file_in and file_out since I have very many lines.inputTTTGGCTTTGTGCAATGG........outputT,T,T,G,G,CT,T,T,GT,G,C,A,A,T,G,GI have used this, but it prints every character vertically: /usr/bin/p

java - gRPC: Generate certificateChainFile and privateKeyFile to make TLS/SSL work

I want to make tls/ssl connection between client and server(Not mutual, one way). This is my setup:Server: Server server = ServerBuilder.forPort(8443) // Enable TLS .useTransportSecurity(certChainFile, privateKeyFile) .addService(new GreetingServiceImpl()) .build();Cli

python - Heap that supports modification of its elements?

Here is my scenario. I want to implement A* (in Python) without having to resort to linear-time min or in operations. I need a heap to be able to efficiently get the lowest weight item. My immediate response was 'Easy! I'll use heapq!' Then I discovered that life is rarely as simple as we would like