Skip to content

commit message#9

Open
MirunaLM wants to merge 24 commits into
masterfrom
lexer
Open

commit message#9
MirunaLM wants to merge 24 commits into
masterfrom
lexer

Conversation

@MirunaLM

@MirunaLM MirunaLM commented Aug 22, 2019

Copy link
Copy Markdown
Collaborator

Description

A lexer is a type of tokenizer that adds a context to the tokens such as the type of token extracted.
A token is composed of data and type, so we have a "Token" class with these attributes
In the "TokenType" enum, we can find our Tokens type like WALK for walk , DIRECTION for North/South etc. To make this tokens we need regular expression(regex).
In the "Lexer" class, the "lex" returns a list of Token objects. This method splits the user's input after space and checks if there is a match with a token from TokenType.

Motivation and Context

Transform the user's input in a list of tokens

How Has This Been Tested?

There are 2 tests: testOutputToken(), which tests if the list of tokens contains a specific token and
testOutputNumberOfWords(), which tests if the number of tokens is equal with the number of tokens words.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@neilcpaul

Copy link
Copy Markdown
Collaborator

Can you do a rebase to pick up the latest changes?

happysad- and others added 6 commits August 23, 2019 12:43
…PREPROD are the same until we use live db, which then will become an environmental secret.
Updated issue templates for issue creation, added request form for requesting access to live servers.
Update issue templates <- creating new issue templates for the future issues.

@martin-bucinskas martin-bucinskas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please review feedback and push the updated code.

@@ -24,15 +23,13 @@ public static ArrayList<Token> lex(String input) {
StringBuffer tokenPatternsBuffer = new StringBuffer();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider changing this to StringBuilder instead of StringBuffer as it will provide better performance.

WALK("walk"), RUN("run"), DIRECTION("North|South|East|West");
public final String pattern;

private TokenType(String pattern) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant private modifier as constructor without a modifier is already a private constructor.

continue;
}
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line and above line indicate something wrongly indented. Check your loop start and end to see if you forgot to indent something.

ArrayList<Token> tokens = Lexer.lex(inputTest);
String output = "";
for (Token token : tokens)
output += token.type + " ";

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of output += token.type + " "; you could use StringBuilder to append strings together and to get the final output you can do sb.toString(). Looks a bit cleaner when dealing with string concatenation.

@@ -41,10 +38,9 @@ public static ArrayList<Token> lex(String input) {
} else if (matcher.group(TokenType.DIRECTION.name()) != null) {
tokens.add(new Token(TokenType.DIRECTION, matcher.group(TokenType.DIRECTION.name())));
continue;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

End of loop and end of if statements there is no need to have continue here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants